howard.objects.variants API documentation

Variants( conn=None, input: str = None, output: str = None, config: dict = {}, param: dict = {}, load: bool = False) View Source

39    def __init__(
40        self,
41        conn=None,
42        input: str = None,
43        output: str = None,
44        config: dict = {},
45        param: dict = {},
46        load: bool = False,
47    ) -> None:
48        """
49        The function `__init__` initializes the variables, sets the input, output, config, param, connexion and
50        header
51
52        :param conn: the connection to the database
53        :param input: the input file
54        :param output: the output file
55        :param config: a dictionary containing the configuration of the model
56        :param param: a dictionary containing the parameters of the model
57        """
58
59        # Init variables
60        self.init_variables()
61
62        # Input
63        self.set_input(input)
64
65        # Config
66        self.set_config(config)
67
68        # Param
69        self.set_param(param)
70
71        # Output
72        self.set_output(output)
73
74        # connexion
75        self.set_connexion(conn)
76
77        # Header
78        self.set_header()
79
80        # Samples
81        self.set_samples()
82
83        # Load data
84        if load:
85            self.load_data()

The function __init__ initializes the variables, sets the input, output, config, param, connexion and header

Parameters

conn: the connection to the database
input: the input file
output: the output file
config: a dictionary containing the configuration of the model
param: a dictionary containing the parameters of the model

def set_samples(self, samples: list = None) -> list: View Source

 87    def set_samples(self, samples: list = None) -> list:
 88        """
 89        The function `set_samples` sets the samples attribute of an object to a provided list or
 90        retrieves it from a parameter dictionary.
 91
 92        :param samples: The `set_samples` method is a method of a class that takes a list of samples as
 93        input and sets the `samples` attribute of the class to the provided list. If no samples are
 94        provided, it tries to get the samples from the class's parameters using the `get_param` method
 95        :type samples: list
 96        :return: The `samples` list is being returned.
 97        """
 98
 99        if not samples:
100            samples = self.get_param().get("samples", {}).get("list", None)
101
102        self.samples = samples
103
104        return samples

The function set_samples sets the samples attribute of an object to a provided list or retrieves it from a parameter dictionary.

Parameters

samples: The set_samples method is a method of a class that takes a list of samples as input and sets the samples attribute of the class to the provided list. If no samples are provided, it tries to get the samples from the class's parameters using the get_param method

Returns

The samples list is being returned.

def get_samples(self) -> list: View Source

106    def get_samples(self) -> list:
107        """
108        This function returns a list of samples.
109        :return: The `get_samples` method is returning the `samples` attribute of the object.
110        """
111
112        return self.samples

This function returns a list of samples.

Returns

The get_samples method is returning the samples attribute of the object.

def get_samples_check(self) -> bool: View Source

114    def get_samples_check(self) -> bool:
115        """
116        This function returns the value of the "check" key within the "samples" dictionary retrieved
117        from the parameters.
118        :return: The method `get_samples_check` is returning the value of the key "check" inside the
119        "samples" dictionary, which is nested inside the dictionary returned by the `get_param()`
120        method. If the key "check" is not found, it will return `False`.
121        """
122
123        return self.get_param().get("samples", {}).get("check", True)

This function returns the value of the "check" key within the "samples" dictionary retrieved from the parameters.

Returns

The method get_samples_check is returning the value of the key "check" inside the "samples" dictionary, which is nested inside the dictionary returned by the get_param() method. If the key "check" is not found, it will return False.

def set_input(self, input: str = None) -> None: View Source

125    def set_input(self, input: str = None) -> None:
126        """
127        The function `set_input` takes a file name as input, extracts the name and extension, and sets
128        attributes in the class accordingly.
129
130        :param input: The `set_input` method in the provided code snippet is used to set attributes
131        related to the input file. Here's a breakdown of the parameters and their usage in the method:
132        :type input: str
133        """
134
135        if input and not isinstance(input, str):
136            try:
137                self.input = input.name
138            except:
139                log.error(f"Input file '{input} in bad format")
140                raise ValueError(f"Input file '{input} in bad format")
141        else:
142            self.input = input
143
144        # Input format
145        if input:
146            input_name, input_extension = os.path.splitext(self.input)
147            self.input_name = input_name
148            self.input_extension = input_extension
149            self.input_format = self.input_extension.replace(".", "")

The function set_input takes a file name as input, extracts the name and extension, and sets attributes in the class accordingly.

Parameters

input: The set_input method in the provided code snippet is used to set attributes related to the input file. Here's a breakdown of the parameters and their usage in the method:

def set_config(self, config: dict) -> None: View Source

151    def set_config(self, config: dict) -> None:
152        """
153        The set_config function takes a config object and assigns it as the configuration object for the
154        class.
155
156        :param config: The `config` parameter in the `set_config` function is a dictionary object that
157        contains configuration settings for the class. When you call the `set_config` function with a
158        dictionary object as the argument, it will set that dictionary as the configuration object for
159        the class
160        :type config: dict
161        """
162
163        self.config = config

The set_config function takes a config object and assigns it as the configuration object for the class.

Parameters

config: The config parameter in the set_config function is a dictionary object that contains configuration settings for the class. When you call the set_config function with a dictionary object as the argument, it will set that dictionary as the configuration object for the class

def set_param(self, param: dict) -> None: View Source

165    def set_param(self, param: dict) -> None:
166        """
167        This function sets a parameter object for the class based on the input dictionary.
168
169        :param param: The `set_param` method you provided takes a dictionary object as input and sets it
170        as the `param` attribute of the class instance
171        :type param: dict
172        """
173
174        self.param = param

This function sets a parameter object for the class based on the input dictionary.

Parameters

param: The set_param method you provided takes a dictionary object as input and sets it as the param attribute of the class instance

def init_variables(self) -> None: View Source

176    def init_variables(self) -> None:
177        """
178        This function initializes the variables that will be used in the rest of the class
179        """
180
181        self.prefix = "howard"
182        self.table_variants = "variants"
183        self.dataframe = None
184
185        self.comparison_map = {
186            "gt": ">",
187            "gte": ">=",
188            "lt": "<",
189            "lte": "<=",
190            "equals": "=",
191            "contains": "SIMILAR TO",
192        }
193
194        self.code_type_map = {"Integer": 0, "String": 1, "Float": 2, "Flag": 3}
195
196        self.code_type_map_to_sql = {
197            "Integer": "INTEGER",
198            "String": "VARCHAR",
199            "Float": "FLOAT",
200            "Flag": "VARCHAR",
201        }
202
203        self.index_additionnal_fields = []

This function initializes the variables that will be used in the rest of the class

def get_indexing(self) -> bool: View Source

205    def get_indexing(self) -> bool:
206        """
207        It returns the value of the key "indexing" in the dictionary. If the key is not present, it
208        returns False.
209        :return: The value of the indexing parameter.
210        """
211
212        return self.get_param().get("indexing", False)

It returns the value of the key "indexing" in the dictionary. If the key is not present, it returns False.

Returns

The value of the indexing parameter.

def get_connexion_config(self) -> dict: View Source

214    def get_connexion_config(self) -> dict:
215        """
216        The function `get_connexion_config` returns a dictionary containing the configuration for a
217        connection, including the number of threads and memory limit.
218        :return: a dictionary containing the configuration for the Connexion library.
219        """
220
221        # config
222        config = self.get_config()
223
224        # Connexion config
225        connexion_config = {}
226        threads = self.get_threads()
227
228        # Threads
229        if threads:
230            connexion_config["threads"] = threads
231
232        # Memory
233        # if config.get("memory", None):
234        #     connexion_config["memory_limit"] = config.get("memory")
235        if self.get_memory():
236            connexion_config["memory_limit"] = self.get_memory()
237
238        # Temporary directory
239        if config.get("tmp", None):
240            connexion_config["temp_directory"] = config.get("tmp")
241
242        # Access
243        if config.get("access", None):
244            access = config.get("access")
245            if access in ["RO"]:
246                access = "READ_ONLY"
247            elif access in ["RW"]:
248                access = "READ_WRITE"
249            connexion_db = self.get_connexion_db()
250            if connexion_db in ":memory:":
251                access = "READ_WRITE"
252            connexion_config["access_mode"] = access
253
254        return connexion_config

The function get_connexion_config returns a dictionary containing the configuration for a connection, including the number of threads and memory limit.

Returns

a dictionary containing the configuration for the Connexion library.

def get_duckdb_settings(self) -> dict: View Source

256    def get_duckdb_settings(self) -> dict:
257        """
258        The function `get_duckdb_settings` retrieves DuckDB settings from a configuration file or a
259        string.
260        :return: The function `get_duckdb_settings` returns a dictionary object `duckdb_settings_dict`.
261        """
262
263        # config
264        config = self.get_config()
265
266        # duckdb settings
267        duckdb_settings_dict = {}
268        if config.get("duckdb_settings", None):
269            duckdb_settings = config.get("duckdb_settings")
270            duckdb_settings = full_path(duckdb_settings)
271            # duckdb setting is a file
272            if os.path.exists(duckdb_settings):
273                with open(duckdb_settings) as json_file:
274                    duckdb_settings_dict = yaml.safe_load(json_file)
275            # duckdb settings is a string
276            else:
277                duckdb_settings_dict = json.loads(duckdb_settings)
278
279        return duckdb_settings_dict

The function get_duckdb_settings retrieves DuckDB settings from a configuration file or a string.

Returns

The function get_duckdb_settings returns a dictionary object duckdb_settings_dict.

def set_connexion_db(self) -> str: View Source

281    def set_connexion_db(self) -> str:
282        """
283        The function `set_connexion_db` returns the appropriate database connection string based on the
284        input format and connection type.
285        :return: the value of the variable `connexion_db`.
286        """
287
288        # Default connexion db
289        default_connexion_db = ":memory:"
290
291        # Find connexion db
292        if self.get_input_format() in ["db", "duckdb"]:
293            connexion_db = self.get_input()
294        elif self.get_connexion_type() in ["memory", default_connexion_db, None]:
295            connexion_db = default_connexion_db
296        elif self.get_connexion_type() in ["tmpfile"]:
297            tmp_name = tempfile.mkdtemp(
298                prefix=self.get_prefix(), dir=self.get_tmp_dir(), suffix=".db"
299            )
300            connexion_db = f"{tmp_name}/tmp.db"
301        elif self.get_connexion_type() != "":
302            connexion_db = self.get_connexion_type()
303        else:
304            connexion_db = default_connexion_db
305
306        # Set connexion db
307        self.connexion_db = connexion_db
308
309        return connexion_db

The function set_connexion_db returns the appropriate database connection string based on the input format and connection type.

Returns

the value of the variable connexion_db.

def set_connexion(self, conn) -> None: View Source

311    def set_connexion(self, conn) -> None:
312        """
313        The function `set_connexion` creates a connection to a database, with options for different
314        database formats and settings.
315
316        :param conn: The `conn` parameter in the `set_connexion` method is the connection to the
317        database. If a connection is not provided, a new connection to an in-memory database is created.
318        The method then proceeds to set up the connection based on the specified format (e.g., duckdb or
319        sqlite
320        """
321
322        # Connexion db
323        connexion_db = self.set_connexion_db()
324
325        # Connexion config
326        connexion_config = self.get_connexion_config()
327
328        # Connexion format
329        connexion_format = self.get_config().get("connexion_format", "duckdb")
330        # Set connexion format
331        self.connexion_format = connexion_format
332
333        # Connexion
334        if not conn:
335            if connexion_format in ["duckdb"]:
336                conn = duckdb.connect(connexion_db, config=connexion_config)
337                # duckDB settings
338                duckdb_settings = self.get_duckdb_settings()
339                if duckdb_settings:
340                    for setting in duckdb_settings:
341                        setting_value = duckdb_settings.get(setting)
342                        if isinstance(setting_value, str):
343                            setting_value = f"'{setting_value}'"
344                        conn.execute(f"PRAGMA {setting}={setting_value};")
345            elif connexion_format in ["sqlite"]:
346                conn = sqlite3.connect(connexion_db)
347
348        # Set connexion
349        self.conn = conn
350
351        # Log
352        log.debug(f"connexion_format: {connexion_format}")
353        log.debug(f"connexion_db: {connexion_db}")
354        log.debug(f"connexion config: {connexion_config}")
355        log.debug(f"connexion duckdb settings: {self.get_duckdb_settings()}")

The function set_connexion creates a connection to a database, with options for different database formats and settings.

Parameters

conn: The conn parameter in the set_connexion method is the connection to the database. If a connection is not provided, a new connection to an in-memory database is created. The method then proceeds to set up the connection based on the specified format (e.g., duckdb or sqlite

def set_output(self, output: str = None) -> None: View Source

357    def set_output(self, output: str = None) -> None:
358        """
359        The `set_output` function in Python sets the output file based on the input or a specified key
360        in the config file, extracting the output name, extension, and format.
361
362        :param output: The `output` parameter in the `set_output` method is used to specify the name of
363        the output file. If the config file has an 'output' key, the method sets the output to the value
364        of that key. If no output is provided, it sets the output to `None`
365        :type output: str
366        """
367
368        if output and not isinstance(output, str):
369            self.output = output.name
370        else:
371            self.output = output
372
373        # Output format
374        if self.output:
375            output_name, output_extension = os.path.splitext(self.output)
376            self.output_name = output_name
377            self.output_extension = output_extension
378            self.output_format = self.output_extension.replace(".", "")
379        else:
380            self.output_name = None
381            self.output_extension = None
382            self.output_format = None

The set_output function in Python sets the output file based on the input or a specified key in the config file, extracting the output name, extension, and format.

Parameters

output: The output parameter in the set_output method is used to specify the name of the output file. If the config file has an 'output' key, the method sets the output to the value of that key. If no output is provided, it sets the output to None

def set_header(self) -> None: View Source

384    def set_header(self) -> None:
385        """
386        It reads the header of a VCF file and stores it as a list of strings and as a VCF object
387        """
388
389        input_file = self.get_input()
390        default_header_list = [
391            "##fileformat=VCFv4.2",
392            "#CHROM	POS	ID	REF	ALT	QUAL	FILTER	INFO",
393        ]
394
395        # Full path
396        input_file = full_path(input_file)
397
398        if input_file:
399
400            input_format = self.get_input_format()
401            input_compressed = self.get_input_compressed()
402            config = self.get_config()
403            header_list = default_header_list
404            if input_format in [
405                "vcf",
406                "hdr",
407                "tsv",
408                "csv",
409                "psv",
410                "parquet",
411                "db",
412                "duckdb",
413            ]:
414                # header provided in param
415                if config.get("header_file", None):
416                    with open(config.get("header_file"), "rt") as f:
417                        header_list = self.read_vcf_header(f)
418                # within a vcf file format (header within input file itsself)
419                elif input_format in ["vcf", "hdr"] and not os.path.isdir(input_file):
420                    # within a compressed vcf file format (.vcf.gz)
421                    if input_compressed:
422                        with bgzf.open(input_file, "rt") as f:
423                            header_list = self.read_vcf_header(f)
424                    # within an uncompressed vcf file format (.vcf)
425                    else:
426                        with open(input_file, "rt") as f:
427                            header_list = self.read_vcf_header(f)
428                # header provided in default external file .hdr
429                elif os.path.exists((input_file + ".hdr")):
430                    with open(input_file + ".hdr", "rt") as f:
431                        header_list = self.read_vcf_header(f)
432                else:
433                    try:  # Try to get header info fields and file columns
434
435                        with tempfile.TemporaryDirectory() as tmpdir:
436
437                            # Create database
438                            db_for_header = Database(database=input_file)
439
440                            # Get header columns for infos fields
441                            db_header_from_columns = (
442                                db_for_header.get_header_from_columns()
443                            )
444
445                            # Get real columns in the file
446                            db_header_columns = db_for_header.get_columns()
447
448                            # Write header file
449                            header_file_tmp = os.path.join(tmpdir, "header")
450                            f = open(header_file_tmp, "w")
451                            vcf.Writer(f, db_header_from_columns)
452                            f.close()
453
454                            # Replace #CHROM line with rel columns
455                            header_list = db_for_header.read_header_file(
456                                header_file=header_file_tmp
457                            )
458                            header_list[-1] = "\t".join(db_header_columns)
459
460                    except:
461
462                        log.warning(
463                            f"No header for file {input_file}. Set as default VCF header"
464                        )
465                        header_list = default_header_list
466
467            else:  # try for unknown format ?
468
469                log.error(f"Input file format '{input_format}' not available")
470                raise ValueError(f"Input file format '{input_format}' not available")
471
472            if not header_list:
473                header_list = default_header_list
474
475            # header as list
476            self.header_list = header_list
477
478            # header as VCF object
479            self.header_vcf = vcf.Reader(io.StringIO("\n".join(header_list)))
480
481        else:
482
483            self.header_list = None
484            self.header_vcf = None

It reads the header of a VCF file and stores it as a list of strings and as a VCF object

def get_query_to_df(self, query: str = '', limit: int = None) -> pandas.core.frame.DataFrame: View Source

486    def get_query_to_df(self, query: str = "", limit: int = None) -> pd.DataFrame:
487        """
488        The `get_query_to_df` function takes a query as a string and returns the result as a pandas
489        DataFrame based on the connection format.
490
491        :param query: The `query` parameter in the `get_query_to_df` function is a string that
492        represents the SQL query you want to execute. This query will be used to fetch data from a
493        database and convert it into a pandas DataFrame
494        :type query: str
495        :param limit: The `limit` parameter in the `get_query_to_df` function is used to specify the
496        maximum number of rows to be returned in the resulting dataframe. If a limit is provided, the
497        function will only fetch up to that number of rows from the database query result. If no limit
498        is specified,
499        :type limit: int
500        :return: A pandas DataFrame is being returned by the `get_query_to_df` function.
501        """
502
503        # Connexion format
504        connexion_format = self.get_connexion_format()
505
506        # Limit in query
507        if limit:
508            pd.set_option("display.max_rows", limit)
509            if connexion_format in ["duckdb"]:
510                df = (
511                    self.conn.execute(query)
512                    .fetch_record_batch(limit)
513                    .read_next_batch()
514                    .to_pandas()
515                )
516            elif connexion_format in ["sqlite"]:
517                df = next(pd.read_sql_query(query, self.conn, chunksize=limit))
518
519        # Full query
520        else:
521            if connexion_format in ["duckdb"]:
522                df = self.conn.execute(query).df()
523            elif connexion_format in ["sqlite"]:
524                df = pd.read_sql_query(query, self.conn)
525
526        return df

The get_query_to_df function takes a query as a string and returns the result as a pandas DataFrame based on the connection format.

Parameters

query: The query parameter in the get_query_to_df function is a string that represents the SQL query you want to execute. This query will be used to fetch data from a database and convert it into a pandas DataFrame
limit: The limit parameter in the get_query_to_df function is used to specify the maximum number of rows to be returned in the resulting dataframe. If a limit is provided, the function will only fetch up to that number of rows from the database query result. If no limit is specified,

Returns

A pandas DataFrame is being returned by the get_query_to_df function.

def get_overview(self) -> None: View Source

528    def get_overview(self) -> None:
529        """
530        The function prints the input, output, config, and dataframe of the current object
531        """
532        table_variants_from = self.get_table_variants(clause="from")
533        sql_columns = self.get_header_columns_as_sql()
534        sql_query_export = f"SELECT {sql_columns} FROM {table_variants_from}"
535        df = self.get_query_to_df(sql_query_export)
536        log.info(
537            "Input:  "
538            + str(self.get_input())
539            + " ["
540            + str(str(self.get_input_format()))
541            + "]"
542        )
543        log.info(
544            "Output: "
545            + str(self.get_output())
546            + " ["
547            + str(str(self.get_output_format()))
548            + "]"
549        )
550        log.info("Config: ")
551        for d in str(json.dumps(self.get_config(), indent=4, sort_keys=True)).split(
552            "\n"
553        ):
554            log.info("\t" + str(d))
555        log.info("Param: ")
556        for d in str(json.dumps(self.get_param(), indent=4, sort_keys=True)).split(
557            "\n"
558        ):
559            log.info("\t" + str(d))
560        log.info("Sample list: " + str(self.get_header_sample_list()))
561        log.info("Dataframe: ")
562        for d in str(df).split("\n"):
563            log.info("\t" + str(d))
564
565        # garbage collector
566        del df
567        gc.collect()
568
569        return None

The function prints the input, output, config, and dataframe of the current object

def get_stats(self) -> dict: View Source

571    def get_stats(self) -> dict:
572        """
573        The `get_stats` function calculates and returns various statistics of the current object,
574        including information about the input file, variants, samples, header fields, quality, and
575        SNVs/InDels.
576        :return: a dictionary containing various statistics of the current object. The dictionary has
577        the following structure:
578        """
579
580        # Log
581        log.info(f"Stats Calculation...")
582
583        # table varaints
584        table_variants_from = self.get_table_variants()
585
586        # stats dict
587        stats = {"Infos": {}}
588
589        ### File
590        input_file = self.get_input()
591        stats["Infos"]["Input file"] = input_file
592
593        # Header
594        header_infos = self.get_header().infos
595        header_formats = self.get_header().formats
596        header_infos_list = list(header_infos)
597        header_formats_list = list(header_formats)
598
599        ### Variants
600
601        stats["Variants"] = {}
602
603        # Variants by chr
604        sql_query_nb_variant_by_chrom = f'SELECT "#CHROM" as CHROM, count(*) as count FROM {table_variants_from} GROUP BY "#CHROM"'
605        df_nb_of_variants_by_chrom = self.get_query_to_df(sql_query_nb_variant_by_chrom)
606        nb_of_variants_by_chrom = df_nb_of_variants_by_chrom.sort_values(
607            by=["CHROM"], kind="quicksort"
608        )
609
610        # Total number of variants
611        nb_of_variants = nb_of_variants_by_chrom["count"].sum()
612
613        # Calculate percentage
614        nb_of_variants_by_chrom["percent"] = nb_of_variants_by_chrom["count"].apply(
615            lambda x: (x / nb_of_variants)
616        )
617
618        stats["Variants"]["Number of variants by chromosome"] = (
619            nb_of_variants_by_chrom.to_dict(orient="index")
620        )
621
622        stats["Infos"]["Number of variants"] = int(nb_of_variants)
623
624        ### Samples
625
626        # Init
627        samples = {}
628        nb_of_samples = 0
629
630        # Check Samples
631        if "GT" in header_formats_list and "FORMAT" in self.get_header_columns():
632            log.debug(f"Check samples...")
633            for sample in self.get_header_sample_list():
634                sql_query_samples = f"""
635                    SELECT  '{sample}' as sample,
636                            REGEXP_EXTRACT("{sample}", '^([0-9/|.]*)[:]*',1) as genotype,
637                            count(REGEXP_EXTRACT("{sample}", '^([0-9/|.]*)[:]*',1)) as count,
638                            concat((count(REGEXP_EXTRACT("{sample}", '^([0-9/|.]*)[:]*',1))/{nb_of_variants})) as percentage
639                    FROM {table_variants_from}
640                    WHERE (
641                        regexp_matches("{sample}", '^[0-9]([/|][0-9])+')
642                        AND
643                        len(string_split(CAST("FORMAT" AS VARCHAR), ':')) = len(string_split(CAST("{sample}" AS VARCHAR), ':'))
644                      )
645                    GROUP BY genotype
646                    """
647                sql_query_genotype_df = self.conn.execute(sql_query_samples).df()
648                sample_genotype_count = sql_query_genotype_df["count"].sum()
649                if len(sql_query_genotype_df):
650                    nb_of_samples += 1
651                    samples[f"{sample} - {sample_genotype_count} variants"] = (
652                        sql_query_genotype_df.to_dict(orient="index")
653                    )
654
655            stats["Samples"] = samples
656            stats["Infos"]["Number of samples"] = nb_of_samples
657
658        # #
659        # if "FORMAT" in self.get_header_columns() and "DP" in header_formats_list:
660        #     stats["Infos"]["Number of samples"] = nb_of_samples
661        # elif nb_of_samples:
662        #     stats["Infos"]["Number of samples"] = "not a VCF format"
663
664        ### INFO and FORMAT fields
665        header_types_df = {}
666        header_types_list = {
667            "List of INFO fields": header_infos,
668            "List of FORMAT fields": header_formats,
669        }
670        i = 0
671        for header_type in header_types_list:
672
673            header_type_infos = header_types_list.get(header_type)
674            header_infos_dict = {}
675
676            for info in header_type_infos:
677
678                i += 1
679                header_infos_dict[i] = {}
680
681                # ID
682                header_infos_dict[i]["id"] = info
683
684                # num
685                genotype_map = {None: ".", -1: "A", -2: "G", -3: "R"}
686                if header_type_infos[info].num in genotype_map.keys():
687                    header_infos_dict[i]["Number"] = genotype_map.get(
688                        header_type_infos[info].num
689                    )
690                else:
691                    header_infos_dict[i]["Number"] = header_type_infos[info].num
692
693                # type
694                if header_type_infos[info].type:
695                    header_infos_dict[i]["Type"] = header_type_infos[info].type
696                else:
697                    header_infos_dict[i]["Type"] = "."
698
699                # desc
700                if header_type_infos[info].desc != None:
701                    header_infos_dict[i]["Description"] = header_type_infos[info].desc
702                else:
703                    header_infos_dict[i]["Description"] = ""
704
705            if len(header_infos_dict):
706                header_types_df[header_type] = pd.DataFrame.from_dict(
707                    header_infos_dict, orient="index"
708                ).to_dict(orient="index")
709
710        # Stats
711        stats["Infos"]["Number of INFO fields"] = len(header_infos_list)
712        stats["Infos"]["Number of FORMAT fields"] = len(header_formats_list)
713        stats["Header"] = header_types_df
714
715        ### QUAL
716        if "QUAL" in self.get_header_columns():
717            sql_query_qual = f"""
718                    SELECT
719                        avg(CAST(QUAL AS INTEGER)) AS Average,
720                        min(CAST(QUAL AS INTEGER)) AS Minimum,
721                        max(CAST(QUAL AS INTEGER)) AS Maximum,
722                        stddev(CAST(QUAL AS INTEGER)) AS StandardDeviation,
723                        median(CAST(QUAL AS INTEGER)) AS Median,
724                        variance(CAST(QUAL AS INTEGER)) AS Variance
725                    FROM {table_variants_from}
726                    WHERE CAST(QUAL AS VARCHAR) NOT IN ('.')
727                    """
728
729            qual = self.conn.execute(sql_query_qual).df().to_dict(orient="index")
730            stats["Quality"] = {"Stats": qual}
731
732        ### SNV and InDel
733
734        sql_query_snv = f"""
735            
736            SELECT Type, count FROM (
737
738                    SELECT
739                        'Total' AS Type,
740                        count(*) AS count
741                    FROM {table_variants_from}
742
743                    UNION
744
745                    SELECT
746                        'MNV' AS Type,
747                        count(*) AS count
748                    FROM {table_variants_from}
749                    WHERE len(REF) > 1 AND len(ALT) > 1
750                    AND len(REF) = len(ALT)
751
752                    UNION
753
754                    SELECT
755                        'InDel' AS Type,
756                        count(*) AS count
757                    FROM {table_variants_from}
758                    WHERE len(REF) > 1 OR len(ALT) > 1
759                    AND len(REF) != len(ALT)
760                    
761                    UNION
762
763                    SELECT
764                        'SNV' AS Type,
765                        count(*) AS count
766                    FROM {table_variants_from}
767                    WHERE len(REF) = 1 AND len(ALT) = 1
768
769                )
770
771            ORDER BY count DESC
772
773                """
774        snv_indel = self.conn.execute(sql_query_snv).df().to_dict(orient="index")
775
776        sql_query_snv_substitution = f"""
777                SELECT
778                    concat(REF, '>', ALT) AS 'Substitution',
779                    count(*) AS count
780                FROM {table_variants_from}
781                WHERE len(REF) = 1 AND len(ALT) = 1
782                GROUP BY REF, ALT
783                ORDER BY count(*) DESC
784                """
785        snv_substitution = (
786            self.conn.execute(sql_query_snv_substitution).df().to_dict(orient="index")
787        )
788        stats["Variants"]["Counts"] = snv_indel
789        stats["Variants"]["Substitutions"] = snv_substitution
790
791        return stats

The get_stats function calculates and returns various statistics of the current object, including information about the input file, variants, samples, header fields, quality, and SNVs/InDels.

Returns

a dictionary containing various statistics of the current object. The dictionary has the following structure:

def stats_to_file(self, file: str = None) -> str: View Source

793    def stats_to_file(self, file: str = None) -> str:
794        """
795        The function `stats_to_file` takes a file name as input, retrieves statistics, serializes them
796        into a JSON object, and writes the JSON object to the specified file.
797
798        :param file: The `file` parameter is a string that represents the file path where the JSON data
799        will be written
800        :type file: str
801        :return: the name of the file that was written to.
802        """
803
804        # Get stats
805        stats = self.get_stats()
806
807        # Serializing json
808        json_object = json.dumps(stats, indent=4)
809
810        # Writing to sample.json
811        with open(file, "w") as outfile:
812            outfile.write(json_object)
813
814        return file

The function stats_to_file takes a file name as input, retrieves statistics, serializes them into a JSON object, and writes the JSON object to the specified file.

Parameters

file: The file parameter is a string that represents the file path where the JSON data will be written

Returns

the name of the file that was written to.

def print_stats(self, output_file: str = None, json_file: str = None) -> None: View Source

816    def print_stats(self, output_file: str = None, json_file: str = None) -> None:
817        """
818        The `print_stats` function generates a markdown file and prints the statistics contained in a
819        JSON file in a formatted manner.
820
821        :param output_file: The `output_file` parameter is a string that specifies the path and filename
822        of the output file where the stats will be printed in Markdown format. If no `output_file` is
823        provided, a temporary directory will be created and the stats will be saved in a file named
824        "stats.md" within that
825        :type output_file: str
826        :param json_file: The `json_file` parameter is a string that represents the path to the JSON
827        file where the statistics will be saved. If no value is provided, a temporary directory will be
828        created and a default file name "stats.json" will be used
829        :type json_file: str
830        :return: The function `print_stats` does not return any value. It has a return type annotation
831        of `None`.
832        """
833
834        # Full path
835        output_file = full_path(output_file)
836        json_file = full_path(json_file)
837
838        with tempfile.TemporaryDirectory() as tmpdir:
839
840            # Files
841            if not output_file:
842                output_file = os.path.join(tmpdir, "stats.md")
843            if not json_file:
844                json_file = os.path.join(tmpdir, "stats.json")
845
846            # Create folders
847            if not os.path.exists(os.path.dirname(output_file)):
848                Path(os.path.dirname(output_file)).mkdir(parents=True, exist_ok=True)
849            if not os.path.exists(os.path.dirname(json_file)):
850                Path(os.path.dirname(json_file)).mkdir(parents=True, exist_ok=True)
851
852            # Create stats JSON file
853            stats_file = self.stats_to_file(file=json_file)
854
855            # Print stats file
856            with open(stats_file) as f:
857                stats = yaml.safe_load(f)
858
859            # Output
860            output_title = []
861            output_index = []
862            output = []
863
864            # Title
865            output_title.append("# HOWARD Stats")
866
867            # Index
868            output_index.append("## Index")
869
870            # Process sections
871            for section in stats:
872                infos = stats.get(section)
873                section_link = "#" + section.lower().replace(" ", "-")
874                output.append(f"## {section}")
875                output_index.append(f"- [{section}]({section_link})")
876
877                if len(infos):
878                    for info in infos:
879                        try:
880                            df = pd.DataFrame.from_dict(infos.get(info), orient="index")
881                            is_df = True
882                        except:
883                            try:
884                                df = pd.DataFrame.from_dict(
885                                    json.loads((infos.get(info))), orient="index"
886                                )
887                                is_df = True
888                            except:
889                                is_df = False
890                        if is_df:
891                            output.append(f"### {info}")
892                            info_link = "#" + info.lower().replace(" ", "-")
893                            output_index.append(f"   - [{info}]({info_link})")
894                            output.append(f"{df.to_markdown(index=False)}")
895                        else:
896                            output.append(f"- {info}: {infos.get(info)}")
897                else:
898                    output.append(f"NA")
899
900            # Write stats in markdown file
901            with open(output_file, "w") as fp:
902                for item in output_title:
903                    fp.write("%s\n" % item)
904                for item in output_index:
905                    fp.write("%s\n" % item)
906                for item in output:
907                    fp.write("%s\n" % item)
908
909            # Output stats in markdown
910            print("")
911            print("\n\n".join(output_title))
912            print("")
913            print("\n\n".join(output))
914            print("")
915
916        return None

The print_stats function generates a markdown file and prints the statistics contained in a JSON file in a formatted manner.

Parameters

output_file: The output_file parameter is a string that specifies the path and filename of the output file where the stats will be printed in Markdown format. If no output_file is provided, a temporary directory will be created and the stats will be saved in a file named "stats.md" within that
json_file: The json_file parameter is a string that represents the path to the JSON file where the statistics will be saved. If no value is provided, a temporary directory will be created and a default file name "stats.json" will be used

Returns

The function print_stats does not return any value. It has a return type annotation of None.

def get_input(self) -> str: View Source

918    def get_input(self) -> str:
919        """
920        It returns the value of the input variable.
921        :return: The input is being returned.
922        """
923        return self.input

It returns the value of the input variable.

Returns

The input is being returned.

def get_input_format(self, input_file: str = None) -> str: View Source

925    def get_input_format(self, input_file: str = None) -> str:
926        """
927        This function returns the format of the input variable, either from the provided input file or
928        by prompting for input.
929
930        :param input_file: The `input_file` parameter in the `get_input_format` method is a string that
931        represents the file path of the input file. If no `input_file` is provided when calling the
932        method, it will default to `None`
933        :type input_file: str
934        :return: The format of the input variable is being returned.
935        """
936
937        if not input_file:
938            input_file = self.get_input()
939        input_format = get_file_format(input_file)
940        return input_format

This function returns the format of the input variable, either from the provided input file or by prompting for input.

Parameters

input_file: The input_file parameter in the get_input_format method is a string that represents the file path of the input file. If no input_file is provided when calling the method, it will default to None

Returns

The format of the input variable is being returned.

def get_input_compressed(self, input_file: str = None) -> str: View Source

942    def get_input_compressed(self, input_file: str = None) -> str:
943        """
944        The function `get_input_compressed` returns the format of the input variable after compressing
945        it.
946
947        :param input_file: The `input_file` parameter in the `get_input_compressed` method is a string
948        that represents the file path of the input file. If no `input_file` is provided when calling the
949        method, it will default to `None` and the method will then call `self.get_input()` to
950        :type input_file: str
951        :return: The function `get_input_compressed` returns the compressed format of the input
952        variable.
953        """
954
955        if not input_file:
956            input_file = self.get_input()
957        input_compressed = get_file_compressed(input_file)
958        return input_compressed

The function get_input_compressed returns the format of the input variable after compressing it.

Parameters

input_file: The input_file parameter in the get_input_compressed method is a string that represents the file path of the input file. If no input_file is provided when calling the method, it will default to None and the method will then call self.get_input() to

Returns

The function get_input_compressed returns the compressed format of the input variable.

def get_output(self) -> str: View Source

960    def get_output(self) -> str:
961        """
962        It returns the output of the neuron.
963        :return: The output of the neural network.
964        """
965
966        return self.output

It returns the output of the neuron.

Returns

The output of the neural network.

def get_output_format(self, output_file: str = None) -> str: View Source

968    def get_output_format(self, output_file: str = None) -> str:
969        """
970        The function `get_output_format` returns the format of the input variable or the output file if
971        provided.
972
973        :param output_file: The `output_file` parameter in the `get_output_format` method is a string
974        that represents the file path of the output file. If no `output_file` is provided when calling
975        the method, it will default to the output obtained from the `get_output` method of the class
976        instance. The
977        :type output_file: str
978        :return: The format of the input variable is being returned.
979        """
980
981        if not output_file:
982            output_file = self.get_output()
983        output_format = get_file_format(output_file)
984
985        return output_format

The function get_output_format returns the format of the input variable or the output file if provided.

Parameters

output_file: The output_file parameter in the get_output_format method is a string that represents the file path of the output file. If no output_file is provided when calling the method, it will default to the output obtained from the get_output method of the class instance. The

Returns

The format of the input variable is being returned.

def get_config(self) -> dict: View Source

987    def get_config(self) -> dict:
988        """
989        It returns the config
990        :return: The config variable is being returned.
991        """
992        return self.config

It returns the config

Returns

The config variable is being returned.

def get_param(self) -> dict: View Source

994    def get_param(self) -> dict:
995        """
996        It returns the param
997        :return: The param variable is being returned.
998        """
999        return self.param

It returns the param

Returns

The param variable is being returned.

def get_connexion_db(self) -> str: View Source

1001    def get_connexion_db(self) -> str:
1002        """
1003        It returns the connexion_db attribute of the object
1004        :return: The connexion_db is being returned.
1005        """
1006        return self.connexion_db

It returns the connexion_db attribute of the object

Returns

The connexion_db is being returned.

def get_prefix(self) -> str: View Source

1008    def get_prefix(self) -> str:
1009        """
1010        It returns the prefix of the object.
1011        :return: The prefix is being returned.
1012        """
1013        return self.prefix

It returns the prefix of the object.

Returns

The prefix is being returned.

def get_table_variants(self, clause: str = 'select') -> str: View Source

1015    def get_table_variants(self, clause: str = "select") -> str:
1016        """
1017        This function returns the table_variants attribute of the object
1018
1019        :param clause: the type of clause the table will be used. Either "select" or "from" (optional),
1020        defaults to select (optional)
1021        :return: The table_variants attribute of the object.
1022        """
1023
1024        # Access
1025        access = self.get_config().get("access", None)
1026
1027        # Clauses "select", "where", "update"
1028        if clause in ["select", "where", "update"]:
1029            table_variants = self.table_variants
1030        # Clause "from"
1031        elif clause in ["from"]:
1032            # For Read Only
1033            if self.get_input_format() in ["parquet"] and access in ["RO"]:
1034                input_file = self.get_input()
1035                table_variants = f"'{input_file}' as variants"
1036            # For Read Write
1037            else:
1038                table_variants = f"{self.table_variants} as variants"
1039        else:
1040            table_variants = self.table_variants
1041        return table_variants

This function returns the table_variants attribute of the object

Parameters

clause: the type of clause the table will be used. Either "select" or "from" (optional), defaults to select (optional)

Returns

The table_variants attribute of the object.

def get_tmp_dir(self) -> str: View Source

1043    def get_tmp_dir(self) -> str:
1044        """
1045        The function `get_tmp_dir` returns the temporary directory path based on configuration
1046        parameters or a default path.
1047        :return: The `get_tmp_dir` method is returning the temporary directory path based on the
1048        configuration, parameters, and a default value of "/tmp".
1049        """
1050
1051        return get_tmp(
1052            config=self.get_config(), param=self.get_param(), default_tmp="/tmp"
1053        )

The function get_tmp_dir returns the temporary directory path based on configuration parameters or a default path.

Returns

The get_tmp_dir method is returning the temporary directory path based on the configuration, parameters, and a default value of "/tmp".

def get_connexion_type(self) -> str: View Source

1055    def get_connexion_type(self) -> str:
1056        """
1057        If the connexion type is not in the list of allowed connexion types, raise a ValueError
1058
1059        :return: The connexion type is being returned.
1060        """
1061        return self.get_config().get("connexion_type", "memory")

If the connexion type is not in the list of allowed connexion types, raise a ValueError

Returns

The connexion type is being returned.

def get_connexion(self): View Source

1063    def get_connexion(self):
1064        """
1065        It returns the connection object
1066
1067        :return: The connection object.
1068        """
1069        return self.conn

It returns the connection object

Returns

The connection object.

def close_connexion(self) -> None: View Source

1071    def close_connexion(self) -> None:
1072        """
1073        This function closes the connection to the database.
1074        :return: The connection is being closed.
1075        """
1076        return self.conn.close()

This function closes the connection to the database.

Returns

The connection is being closed.

def get_header(self, type: str = 'vcf'): View Source

1078    def get_header(self, type: str = "vcf"):
1079        """
1080        This function returns the header of the VCF file as a list of strings
1081
1082        :param type: the type of header you want to get, defaults to vcf (optional)
1083        :return: The header of the vcf file.
1084        """
1085
1086        if self.header_vcf:
1087            if type == "vcf":
1088                return self.header_vcf
1089            elif type == "list":
1090                return self.header_list
1091        else:
1092            if type == "vcf":
1093                header = vcf.Reader(io.StringIO("\n".join(vcf_required)))
1094                return header
1095            elif type == "list":
1096                return vcf_required

This function returns the header of the VCF file as a list of strings

Parameters

type: the type of header you want to get, defaults to vcf (optional)

Returns

The header of the vcf file.

def get_header_infos_list(self) -> list: View Source

1098    def get_header_infos_list(self) -> list:
1099        """
1100        This function retrieves a list of information fields from the header.
1101        :return: A list of information fields from the header.
1102        """
1103
1104        # Init
1105        infos_list = []
1106
1107        for field in self.get_header().infos:
1108            infos_list.append(field)
1109
1110        return infos_list

This function retrieves a list of information fields from the header.

Returns

A list of information fields from the header.

def get_header_length(self, file: str = None) -> int: View Source

1112    def get_header_length(self, file: str = None) -> int:
1113        """
1114        The function `get_header_length` returns the length of the header list, excluding the #CHROM
1115        line.
1116
1117        :param file: The `file` parameter is an optional argument that specifies the path to a VCF
1118        header file. If this argument is provided, the function will read the header from the specified
1119        file and return the length of the header list minus 1 (to exclude the #CHROM line)
1120        :type file: str
1121        :return: the length of the header list, excluding the #CHROM line.
1122        """
1123
1124        if file:
1125            return len(self.read_vcf_header_file(file=file)) - 1
1126        elif self.get_header(type="list"):
1127            return len(self.get_header(type="list")) - 1
1128        else:
1129            return 0

The function get_header_length returns the length of the header list, excluding the #CHROM line.

Parameters

file: The file parameter is an optional argument that specifies the path to a VCF header file. If this argument is provided, the function will read the header from the specified file and return the length of the header list minus 1 (to exclude the #CHROM line)

Returns

the length of the header list, excluding the #CHROM line.

def get_header_columns(self) -> str: View Source

1131    def get_header_columns(self) -> str:
1132        """
1133        This function returns the header list of a VCF
1134
1135        :return: The length of the header list.
1136        """
1137        if self.get_header():
1138            return self.get_header(type="list")[-1]
1139        else:
1140            return ""

This function returns the header list of a VCF

Returns

The length of the header list.

def get_header_columns_as_list(self) -> list: View Source

1142    def get_header_columns_as_list(self) -> list:
1143        """
1144        This function returns the header list of a VCF
1145
1146        :return: The length of the header list.
1147        """
1148        if self.get_header():
1149            return self.get_header_columns().strip().split("\t")
1150        else:
1151            return []

This function returns the header list of a VCF

Returns

The length of the header list.

def get_header_columns_as_sql(self) -> str: View Source

1153    def get_header_columns_as_sql(self) -> str:
1154        """
1155        This function retruns header length (without #CHROM line)
1156
1157        :return: The length of the header list.
1158        """
1159        sql_column_list = []
1160        for col in self.get_header_columns_as_list():
1161            sql_column_list.append(f'"{col}"')
1162        return ",".join(sql_column_list)

This function retruns header length (without #CHROM line)

Returns

The length of the header list.

def get_header_sample_list( self, check: bool = False, samples: list = None, samples_force: bool = False) -> list: View Source

1164    def get_header_sample_list(
1165        self, check: bool = False, samples: list = None, samples_force: bool = False
1166    ) -> list:
1167        """
1168        The function `get_header_sample_list` returns a list of samples from a VCF header, with optional
1169        checking and filtering based on input parameters.
1170
1171        :param check: The `check` parameter in the `get_header_sample_list` function is a boolean
1172        parameter that determines whether to check if the samples in the list are properly defined as
1173        genotype columns. If `check` is set to `True`, the function will verify if each sample in the
1174        list is defined as a, defaults to False
1175        :type check: bool (optional)
1176        :param samples: The `samples` parameter in the `get_header_sample_list` function is a list that
1177        allows you to specify a subset of samples from the header. If you provide a list of sample
1178        names, the function will check if each sample is defined in the header. If a sample is not found
1179        in the
1180        :type samples: list
1181        :param samples_force: The `samples_force` parameter in the `get_header_sample_list` function is
1182        a boolean parameter that determines whether to force the function to return the sample list
1183        without checking if the samples are genotype columns. If `samples_force` is set to `True`, the
1184        function will return the sample list without performing, defaults to False
1185        :type samples_force: bool (optional)
1186        :return: The function `get_header_sample_list` returns a list of samples based on the input
1187        parameters and conditions specified in the function.
1188        """
1189
1190        # Init
1191        samples_list = []
1192
1193        if samples is None:
1194            samples_list = self.header_vcf.samples
1195        else:
1196            samples_checked = []
1197            for sample in samples:
1198                if sample in self.header_vcf.samples:
1199                    samples_checked.append(sample)
1200                else:
1201                    log.warning(f"Sample '{sample}' not defined in header")
1202            samples_list = samples_checked
1203
1204            # Force sample list without checking if is_genotype_column
1205            if samples_force:
1206                log.warning(f"Samples {samples_list} not checked if genotypes")
1207                return samples_list
1208
1209        if check:
1210            samples_checked = []
1211            for sample in samples_list:
1212                if self.is_genotype_column(column=sample):
1213                    samples_checked.append(sample)
1214                else:
1215                    log.warning(
1216                        f"Sample '{sample}' not defined as a sample (genotype not well defined)"
1217                    )
1218            samples_list = samples_checked
1219
1220        # Return samples list
1221        return samples_list

The function get_header_sample_list returns a list of samples from a VCF header, with optional checking and filtering based on input parameters.

Parameters

check: The check parameter in the get_header_sample_list function is a boolean parameter that determines whether to check if the samples in the list are properly defined as genotype columns. If check is set to True, the function will verify if each sample in the list is defined as a, defaults to False
samples: The samples parameter in the get_header_sample_list function is a list that allows you to specify a subset of samples from the header. If you provide a list of sample names, the function will check if each sample is defined in the header. If a sample is not found in the
samples_force: The samples_force parameter in the get_header_sample_list function is a boolean parameter that determines whether to force the function to return the sample list without checking if the samples are genotype columns. If samples_force is set to True, the function will return the sample list without performing, defaults to False

Returns

The function get_header_sample_list returns a list of samples based on the input parameters and conditions specified in the function.

def is_genotype_column(self, column: str = None) -> bool: View Source

1223    def is_genotype_column(self, column: str = None) -> bool:
1224        """
1225        This function checks if a given column is a genotype column in a database.
1226
1227        :param column: The `column` parameter in the `is_genotype_column` method is a string that
1228        represents the column name in a database table. This method checks if the specified column is a
1229        genotype column in the database. If a column name is provided, it calls the `is_genotype_column`
1230        method of
1231        :type column: str
1232        :return: The `is_genotype_column` method is returning a boolean value. If the `column` parameter
1233        is not None, it calls the `is_genotype_column` method of the `Database` class with the specified
1234        column name and returns the result. If the `column` parameter is None, it returns False.
1235        """
1236
1237        if column is not None:
1238            return Database(database=self.get_input()).is_genotype_column(column=column)
1239        else:
1240            return False

This function checks if a given column is a genotype column in a database.

Parameters

column: The column parameter in the is_genotype_column method is a string that represents the column name in a database table. This method checks if the specified column is a genotype column in the database. If a column name is provided, it calls the is_genotype_column method of

Returns

The is_genotype_column method is returning a boolean value. If the column parameter is not None, it calls the is_genotype_column method of the Database class with the specified column name and returns the result. If the column parameter is None, it returns False.

def get_verbose(self) -> bool: View Source

1242    def get_verbose(self) -> bool:
1243        """
1244        It returns the value of the "verbose" key in the config dictionary, or False if the key doesn't
1245        exist
1246
1247        :return: The value of the key "verbose" in the config dictionary.
1248        """
1249        return self.get_config().get("verbose", False)

It returns the value of the "verbose" key in the config dictionary, or False if the key doesn't exist

Returns

The value of the key "verbose" in the config dictionary.

def get_connexion_format(self) -> str: View Source

1251    def get_connexion_format(self) -> str:
1252        """
1253        It returns the connexion format of the object.
1254        :return: The connexion_format is being returned.
1255        """
1256        connexion_format = self.connexion_format
1257        if connexion_format not in ["duckdb", "sqlite"]:
1258            log.error(f"Unknown connexion format {connexion_format}")
1259            raise ValueError(f"Unknown connexion format {connexion_format}")
1260        else:
1261            return connexion_format

It returns the connexion format of the object.

Returns

The connexion_format is being returned.

def insert_file_to_table( self, file, columns: str, header_len: int = 0, sep: str = '\t', chunksize: int = 1000000) -> None: View Source

1263    def insert_file_to_table(
1264        self,
1265        file,
1266        columns: str,
1267        header_len: int = 0,
1268        sep: str = "\t",
1269        chunksize: int = 1000000,
1270    ) -> None:
1271        """
1272        The function reads a file in chunks and inserts each chunk into a table based on the specified
1273        database format.
1274
1275        :param file: The `file` parameter is the file that you want to load into a table. It should be
1276        the path to the file on your system
1277        :param columns: The `columns` parameter in the `insert_file_to_table` function is a string that
1278        should contain the names of the columns in the table where the data will be inserted. The column
1279        names should be separated by commas within the string. For example, if you have columns named
1280        "id", "name
1281        :type columns: str
1282        :param header_len: The `header_len` parameter in the `insert_file_to_table` function specifies
1283        the number of lines to skip at the beginning of the file before reading the actual data. This
1284        parameter allows you to skip any header information present in the file before processing the
1285        data, defaults to 0
1286        :type header_len: int (optional)
1287        :param sep: The `sep` parameter in the `insert_file_to_table` function is used to specify the
1288        separator character that is used in the file being read. In this case, the default separator is
1289        set to `\t`, which represents a tab character. You can change this parameter to a different
1290        separator character if, defaults to \t
1291        :type sep: str (optional)
1292        :param chunksize: The `chunksize` parameter specifies the number of rows to read in at a time
1293        when processing the file in chunks. In the provided code snippet, the default value for
1294        `chunksize` is set to 1000000. This means that the file will be read in chunks of 1,, defaults
1295        to 1000000
1296        :type chunksize: int (optional)
1297        """
1298
1299        # Config
1300        chunksize = self.get_config().get("load", {}).get("chunk", chunksize)
1301        connexion_format = self.get_connexion_format()
1302
1303        log.debug("chunksize: " + str(chunksize))
1304
1305        if chunksize:
1306            for chunk in pd.read_csv(
1307                file, skiprows=header_len, sep=sep, chunksize=chunksize, engine="c"
1308            ):
1309                if connexion_format in ["duckdb"]:
1310                    sql_insert_into = (
1311                        f"INSERT INTO variants ({columns}) SELECT {columns} FROM chunk"
1312                    )
1313                    self.conn.execute(sql_insert_into)
1314                elif connexion_format in ["sqlite"]:
1315                    chunk.to_sql("variants", self.conn, if_exists="append", index=False)

The function reads a file in chunks and inserts each chunk into a table based on the specified database format.

Parameters

file: The file parameter is the file that you want to load into a table. It should be the path to the file on your system
columns: The columns parameter in the insert_file_to_table function is a string that should contain the names of the columns in the table where the data will be inserted. The column names should be separated by commas within the string. For example, if you have columns named "id", "name
header_len: The header_len parameter in the insert_file_to_table function specifies the number of lines to skip at the beginning of the file before reading the actual data. This parameter allows you to skip any header information present in the file before processing the data, defaults to 0
sep: The sep parameter in the insert_file_to_table function is used to specify the separator character that is used in the file being read. In this case, the default separator is set to , which represents a tab character. You can change this parameter to a different separator character if, defaults to
chunksize: The chunksize parameter specifies the number of rows to read in at a time when processing the file in chunks. In the provided code snippet, the default value for chunksize is set to 1000000. This means that the file will be read in chunks of 1,, defaults to 1000000

def load_data( self, input_file: str = None, drop_variants_table: bool = False, sample_size: int = 20480) -> None: View Source

1317    def load_data(
1318        self,
1319        input_file: str = None,
1320        drop_variants_table: bool = False,
1321        sample_size: int = 20480,
1322    ) -> None:
1323        """
1324        The `load_data` function reads a VCF file and inserts it into a table, with options to drop the
1325        table before loading the data and specify a sample size.
1326
1327        :param input_file: The path to the input file. This is the VCF file that will be loaded into the
1328        table
1329        :type input_file: str
1330        :param drop_variants_table: The `drop_variants_table` parameter is a boolean flag that
1331        determines whether the variants table should be dropped before loading the data. If set to
1332        `True`, the variants table will be dropped. If set to `False` (default), the variants table will
1333        not be dropped, defaults to False
1334        :type drop_variants_table: bool (optional)
1335        :param sample_size: The `sample_size` parameter determines the number of rows to be sampled from
1336        the input file. If it is set to `None`, the default value of 20480 will be used, defaults to
1337        20480
1338        :type sample_size: int (optional)
1339        """
1340
1341        log.info("Loading...")
1342
1343        # change input file
1344        if input_file:
1345            self.set_input(input_file)
1346            self.set_header()
1347
1348        # drop variants table
1349        if drop_variants_table:
1350            self.drop_variants_table()
1351
1352        # get table variants
1353        table_variants = self.get_table_variants()
1354
1355        # Access
1356        access = self.get_config().get("access", None)
1357        log.debug(f"access: {access}")
1358
1359        # Input format and compress
1360        input_format = self.get_input_format()
1361        input_compressed = self.get_input_compressed()
1362        log.debug(f"input_format: {input_format}")
1363        log.debug(f"input_compressed: {input_compressed}")
1364
1365        # input_compressed_format
1366        if input_compressed:
1367            input_compressed_format = "gzip"
1368        else:
1369            input_compressed_format = "none"
1370        log.debug(f"input_compressed_format: {input_compressed_format}")
1371
1372        # Connexion format
1373        connexion_format = self.get_connexion_format()
1374
1375        # Sample size
1376        if not sample_size:
1377            sample_size = -1
1378        log.debug(f"sample_size: {sample_size}")
1379
1380        # Load data
1381        log.debug(f"Load Data from {input_format}")
1382
1383        # DuckDB connexion
1384        if connexion_format in ["duckdb"]:
1385
1386            # Database already exists
1387            if self.input_format in ["db", "duckdb"]:
1388
1389                if connexion_format in ["duckdb"]:
1390                    log.debug(f"Input file format '{self.input_format}' duckDB")
1391                else:
1392                    log.error(
1393                        f"Input file format '{self.input_format}' not compatilbe with database format '{connexion_format}'"
1394                    )
1395                    raise ValueError(
1396                        f"Input file format '{self.input_format}' not compatilbe with database format '{connexion_format}'"
1397                    )
1398
1399            # Load from existing database format
1400            else:
1401
1402                try:
1403                    # Create Table or View
1404                    database = Database(database=self.input)
1405                    sql_from = database.get_sql_from(sample_size=sample_size)
1406
1407                    if access in ["RO"]:
1408                        sql_load = (
1409                            f"CREATE VIEW {table_variants} AS SELECT * FROM {sql_from}"
1410                        )
1411                    else:
1412                        sql_load = (
1413                            f"CREATE TABLE {table_variants} AS SELECT * FROM {sql_from}"
1414                        )
1415                    self.conn.execute(sql_load)
1416
1417                except:
1418                    # Format not available
1419                    log.error(f"Input file format '{self.input_format}' not available")
1420                    raise ValueError(
1421                        f"Input file format '{self.input_format}' not available"
1422                    )
1423
1424        # SQLite connexion
1425        elif connexion_format in ["sqlite"] and input_format in [
1426            "vcf",
1427            "tsv",
1428            "csv",
1429            "psv",
1430        ]:
1431
1432            # Main structure
1433            structure = {
1434                "#CHROM": "VARCHAR",
1435                "POS": "INTEGER",
1436                "ID": "VARCHAR",
1437                "REF": "VARCHAR",
1438                "ALT": "VARCHAR",
1439                "QUAL": "VARCHAR",
1440                "FILTER": "VARCHAR",
1441                "INFO": "VARCHAR",
1442            }
1443
1444            # Strcuture with samples
1445            structure_complete = structure
1446            if self.get_header_sample_list():
1447                structure["FORMAT"] = "VARCHAR"
1448                for sample in self.get_header_sample_list():
1449                    structure_complete[sample] = "VARCHAR"
1450
1451            # Columns list for create and insert
1452            sql_create_table_columns = []
1453            sql_create_table_columns_list = []
1454            for column in structure_complete:
1455                column_type = structure_complete[column]
1456                sql_create_table_columns.append(
1457                    f'"{column}" {column_type} default NULL'
1458                )
1459                sql_create_table_columns_list.append(f'"{column}"')
1460
1461            # Create database
1462            log.debug(f"Create Table {table_variants}")
1463            sql_create_table_columns_sql = ", ".join(sql_create_table_columns)
1464            sql_create_table_columns_list_sql = ", ".join(sql_create_table_columns_list)
1465            sql_create_table = f"CREATE TABLE IF NOT EXISTS {table_variants} ({sql_create_table_columns_sql})"
1466            self.conn.execute(sql_create_table)
1467
1468            # chunksize define length of file chunk load file
1469            chunksize = 100000
1470
1471            # delimiter
1472            delimiter = file_format_delimiters.get(input_format, "\t")
1473
1474            # Load the input file
1475            with open(self.input, "rt") as input_file:
1476
1477                # Use the appropriate file handler based on the input format
1478                if input_compressed:
1479                    input_file = bgzf.open(self.input, "rt")
1480                if input_format in ["vcf"]:
1481                    header_len = self.get_header_length()
1482                else:
1483                    header_len = 0
1484
1485                # Insert the file contents into a table
1486                self.insert_file_to_table(
1487                    input_file,
1488                    columns=sql_create_table_columns_list_sql,
1489                    header_len=header_len,
1490                    sep=delimiter,
1491                    chunksize=chunksize,
1492                )
1493
1494        else:
1495            log.error(
1496                f"Connexion format '{connexion_format}' not available with format '{input_format}'"
1497            )
1498            raise ValueError(
1499                f"Connexion format '{connexion_format}' not available with format '{input_format}'"
1500            )
1501
1502        # Explode INFOS fields into table fields
1503        if self.get_explode_infos():
1504            self.explode_infos(
1505                prefix=self.get_explode_infos_prefix(),
1506                fields=self.get_explode_infos_fields(),
1507                force=True,
1508            )
1509
1510        # Create index after insertion
1511        self.create_indexes()

The load_data function reads a VCF file and inserts it into a table, with options to drop the table before loading the data and specify a sample size.

Parameters

input_file: The path to the input file. This is the VCF file that will be loaded into the table
drop_variants_table: The drop_variants_table parameter is a boolean flag that determines whether the variants table should be dropped before loading the data. If set to True, the variants table will be dropped. If set to False (default), the variants table will not be dropped, defaults to False
sample_size: The sample_size parameter determines the number of rows to be sampled from the input file. If it is set to None, the default value of 20480 will be used, defaults to 20480

def get_explode_infos(self) -> bool: View Source

1513    def get_explode_infos(self) -> bool:
1514        """
1515        The function `get_explode_infos` returns the value of the "explode_infos" parameter, defaulting
1516        to False if it is not set.
1517        :return: The method is returning the value of the "explode_infos" parameter, which is a boolean
1518        value. If the parameter is not present, it will return False.
1519        """
1520
1521        return self.get_param().get("explode", {}).get("explode_infos", False)

The function get_explode_infos returns the value of the "explode_infos" parameter, defaulting to False if it is not set.

Returns

The method is returning the value of the "explode_infos" parameter, which is a boolean value. If the parameter is not present, it will return False.

def get_explode_infos_fields( self, explode_infos_fields: str = None, remove_fields_not_in_header: bool = False) -> list: View Source

1523    def get_explode_infos_fields(
1524        self,
1525        explode_infos_fields: str = None,
1526        remove_fields_not_in_header: bool = False,
1527    ) -> list:
1528        """
1529        The `get_explode_infos_fields` function returns a list of exploded information fields based on
1530        the input parameter `explode_infos_fields`.
1531
1532        :param explode_infos_fields: The `explode_infos_fields` parameter is a string that specifies the
1533        fields to be exploded. It can be set to "ALL" to explode all fields, or it can be a
1534        comma-separated list of field names to explode
1535        :type explode_infos_fields: str
1536        :param remove_fields_not_in_header: The parameter `remove_fields_not_in_header` is a boolean
1537        flag that determines whether to remove fields that are not present in the header. If it is set
1538        to `True`, any field that is not in the header will be excluded from the list of exploded
1539        information fields. If it is set to `, defaults to False
1540        :type remove_fields_not_in_header: bool (optional)
1541        :return: The function `get_explode_infos_fields` returns a list of exploded information fields.
1542        If the `explode_infos_fields` parameter is not provided or is set to None, it returns an empty
1543        list. If the parameter is provided and its value is "ALL", it also returns an empty list.
1544        Otherwise, it returns a list of exploded information fields after removing any spaces and
1545        splitting the string by commas.
1546        """
1547
1548        # If no fields, get it in param
1549        if not explode_infos_fields:
1550            explode_infos_fields = (
1551                self.get_param().get("explode", {}).get("explode_infos_fields", None)
1552            )
1553
1554        # If no fields, defined as all fields in header using keyword
1555        if not explode_infos_fields:
1556            explode_infos_fields = "*"
1557
1558        # If fields list not empty
1559        if explode_infos_fields:
1560
1561            # Input fields list
1562            if isinstance(explode_infos_fields, str):
1563                fields_input = explode_infos_fields.split(",")
1564            elif isinstance(explode_infos_fields, list):
1565                fields_input = explode_infos_fields
1566            else:
1567                fields_input = []
1568
1569            # Fields list without * keyword
1570            fields_without_all = fields_input.copy()
1571            if "*".casefold() in (item.casefold() for item in fields_without_all):
1572                fields_without_all.remove("*")
1573
1574            # Fields in header
1575            fields_in_header = sorted(list(set(self.get_header().infos)))
1576
1577            # Construct list of fields
1578            fields_output = []
1579            for field in fields_input:
1580
1581                # Strip field
1582                field = field.strip()
1583
1584                # format keyword * in regex
1585                if field.upper() in ["*"]:
1586                    field = ".*"
1587
1588                # Find all fields with pattern
1589                r = re.compile(rf"^{field}$")
1590                fields_search = sorted(list(filter(r.match, fields_in_header)))
1591
1592                # Remove fields input from search
1593                if field in fields_search:
1594                    fields_search = [field]
1595                elif fields_search != [field]:
1596                    fields_search = sorted(
1597                        list(set(fields_search).difference(fields_input))
1598                    )
1599
1600                # If field is not in header (avoid not well formatted header)
1601                if not fields_search and not remove_fields_not_in_header:
1602                    fields_search = [field]
1603
1604                # Add found fields
1605                for new_field in fields_search:
1606                    # Add field, if not already exists, and if it is in header (if asked)
1607                    if (
1608                        new_field not in fields_output
1609                        and (
1610                            not remove_fields_not_in_header
1611                            or new_field in fields_in_header
1612                        )
1613                        and new_field not in [".*"]
1614                    ):
1615                        fields_output.append(new_field)
1616
1617            return fields_output
1618
1619        else:
1620
1621            return []

The get_explode_infos_fields function returns a list of exploded information fields based on the input parameter explode_infos_fields.

Parameters

explode_infos_fields: The explode_infos_fields parameter is a string that specifies the fields to be exploded. It can be set to "ALL" to explode all fields, or it can be a comma-separated list of field names to explode
remove_fields_not_in_header: The parameter remove_fields_not_in_header is a boolean flag that determines whether to remove fields that are not present in the header. If it is set to True, any field that is not in the header will be excluded from the list of exploded information fields. If it is set to `, defaults to False

Returns

The function get_explode_infos_fields returns a list of exploded information fields. If the explode_infos_fields parameter is not provided or is set to None, it returns an empty list. If the parameter is provided and its value is "ALL", it also returns an empty list. Otherwise, it returns a list of exploded information fields after removing any spaces and splitting the string by commas.

def get_explode_infos_prefix(self, explode_infos_prefix: str = None) -> str: View Source

1623    def get_explode_infos_prefix(self, explode_infos_prefix: str = None) -> str:
1624        """
1625        The function `get_explode_infos_prefix` returns the value of the `explode_infos_prefix` parameter, or
1626        the value of `self.get_param().get("explode_infos_prefix", None)` if `explode_infos_prefix` is
1627        not provided.
1628
1629        :param explode_infos_prefix: The parameter `explode_infos_prefix` is a string that specifies a
1630        prefix to be used for exploding or expanding information
1631        :type explode_infos_prefix: str
1632        :return: the value of the variable `explode_infos_prefix`.
1633        """
1634
1635        if not explode_infos_prefix:
1636            explode_infos_prefix = (
1637                self.get_param().get("explode", {}).get("explode_infos_prefix", "")
1638            )
1639
1640        return explode_infos_prefix

The function get_explode_infos_prefix returns the value of the explode_infos_prefix parameter, or the value of self.get_param().get("explode_infos_prefix", None) if explode_infos_prefix is not provided.

Parameters

explode_infos_prefix: The parameter explode_infos_prefix is a string that specifies a prefix to be used for exploding or expanding information

Returns

the value of the variable explode_infos_prefix.

def add_column( self, table_name, column_name, column_type, default_value=None, drop: bool = False) -> dict: View Source

1642    def add_column(
1643        self,
1644        table_name,
1645        column_name,
1646        column_type,
1647        default_value=None,
1648        drop: bool = False,
1649    ) -> dict:
1650        """
1651        The `add_column` function adds a column to a SQLite or DuckDB table with a default value if it
1652        doesn't already exist.
1653
1654        :param table_name: The name of the table to which you want to add a column
1655        :param column_name: The parameter "column_name" is the name of the column that you want to add
1656        to the table
1657        :param column_type: The `column_type` parameter specifies the data type of the column that you
1658        want to add to the table. It should be a string that represents the desired data type, such as
1659        "INTEGER", "TEXT", "REAL", etc
1660        :param default_value: The `default_value` parameter is an optional parameter that specifies the
1661        default value for the newly added column. If a default value is provided, it will be assigned to
1662        the column for any existing rows that do not have a value for that column
1663        :param drop: The `drop` parameter is a boolean flag that determines whether to drop the column
1664        if it already exists in the table. If `drop` is set to `True`, the function will drop the
1665        existing column before adding the new column. If `drop` is set to `False` (default),, defaults
1666        to False
1667        :type drop: bool (optional)
1668        :return: a boolean value indicating whether the column was successfully added to the table.
1669        """
1670
1671        # added
1672        added = False
1673        dropped = False
1674
1675        # Check if the column already exists in the table
1676        query = f""" SELECT * FROM {table_name} LIMIT 0 """
1677        columns = self.get_query_to_df(query).columns.tolist()
1678        if column_name.upper() in [c.upper() for c in columns]:
1679            log.debug(
1680                f"The {column_name} column already exists in the {table_name} table"
1681            )
1682            if drop:
1683                self.drop_column(table_name=table_name, column_name=column_name)
1684                dropped = True
1685            else:
1686                return None
1687        else:
1688            log.debug(f"The {column_name} column NOT exists in the {table_name} table")
1689
1690        # Add column in table
1691        add_column_query = (
1692            f""" ALTER TABLE {table_name} ADD COLUMN "{column_name}" {column_type} """
1693        )
1694        if default_value is not None:
1695            add_column_query += f" DEFAULT {default_value}"
1696        self.execute_query(add_column_query)
1697        added = not dropped
1698        log.debug(
1699            f"The {column_name} column was successfully added to the {table_name} table"
1700        )
1701
1702        if added:
1703            added_column = {
1704                "table_name": table_name,
1705                "column_name": column_name,
1706                "column_type": column_type,
1707                "default_value": default_value,
1708            }
1709        else:
1710            added_column = None
1711
1712        return added_column

The add_column function adds a column to a SQLite or DuckDB table with a default value if it doesn't already exist.

Parameters

table_name: The name of the table to which you want to add a column
column_name: The parameter "column_name" is the name of the column that you want to add to the table
column_type: The column_type parameter specifies the data type of the column that you want to add to the table. It should be a string that represents the desired data type, such as "INTEGER", "TEXT", "REAL", etc
default_value: The default_value parameter is an optional parameter that specifies the default value for the newly added column. If a default value is provided, it will be assigned to the column for any existing rows that do not have a value for that column
drop: The drop parameter is a boolean flag that determines whether to drop the column if it already exists in the table. If drop is set to True, the function will drop the existing column before adding the new column. If drop is set to False (default),, defaults to False

Returns

a boolean value indicating whether the column was successfully added to the table.

def drop_column( self, column: dict = None, table_name: str = None, column_name: str = None) -> bool: View Source

1714    def drop_column(
1715        self, column: dict = None, table_name: str = None, column_name: str = None
1716    ) -> bool:
1717        """
1718        The `drop_column` function drops a specified column from a given table in a database and returns
1719        True if the column was successfully dropped, and False if the column does not exist in the
1720        table.
1721
1722        :param column: The `column` parameter is a dictionary that contains information about the column
1723        you want to drop. It has two keys:
1724        :type column: dict
1725        :param table_name: The `table_name` parameter is the name of the table from which you want to
1726        drop a column
1727        :type table_name: str
1728        :param column_name: The `column_name` parameter is the name of the column that you want to drop
1729        from the table
1730        :type column_name: str
1731        :return: a boolean value. It returns True if the column was successfully dropped from the table,
1732        and False if the column does not exist in the table.
1733        """
1734
1735        # Find column infos
1736        if column:
1737            if isinstance(column, dict):
1738                table_name = column.get("table_name", None)
1739                column_name = column.get("column_name", None)
1740            elif isinstance(column, str):
1741                table_name = self.get_table_variants()
1742                column_name = column
1743            else:
1744                table_name = None
1745                column_name = None
1746
1747        if not table_name and not column_name:
1748            return False
1749
1750        # Removed
1751        removed = False
1752
1753        # Check if the column already exists in the table
1754        query = f""" SELECT * FROM {table_name} LIMIT 0 """
1755        columns = self.get_query_to_df(query).columns.tolist()
1756        if column_name in columns:
1757            log.debug(f"The {column_name} column exists in the {table_name} table")
1758        else:
1759            log.debug(f"The {column_name} column NOT exists in the {table_name} table")
1760            return False
1761
1762        # Add column in table # ALTER TABLE integers DROP k
1763        add_column_query = f""" ALTER TABLE {table_name} DROP "{column_name}" """
1764        self.execute_query(add_column_query)
1765        removed = True
1766        log.debug(
1767            f"The {column_name} column was successfully dropped to the {table_name} table"
1768        )
1769
1770        return removed

The drop_column function drops a specified column from a given table in a database and returns True if the column was successfully dropped, and False if the column does not exist in the table.

Parameters

column: The column parameter is a dictionary that contains information about the column you want to drop. It has two keys:
table_name: The table_name parameter is the name of the table from which you want to drop a column
column_name: The column_name parameter is the name of the column that you want to drop from the table

Returns

a boolean value. It returns True if the column was successfully dropped from the table, and False if the column does not exist in the table.

def explode_infos( self, prefix: str = None, create_index: bool = False, fields: list = None, force: bool = False, proccess_all_fields_together: bool = False, table: str = None) -> list: View Source

1772    def explode_infos(
1773        self,
1774        prefix: str = None,
1775        create_index: bool = False,
1776        fields: list = None,
1777        force: bool = False,
1778        proccess_all_fields_together: bool = False,
1779        table: str = None,
1780    ) -> list:
1781        """
1782        The `explode_infos` function in Python takes a VCF file and explodes the INFO fields into
1783        individual columns, returning a list of added columns.
1784
1785        :param prefix: The `prefix` parameter is a string that is used as a prefix for the exploded INFO
1786        fields. If the `prefix` is not provided or is set to `None`, the function will use the value of
1787        `self.get_explode_infos_prefix()` as the prefix
1788        :type prefix: str
1789        :param create_index: The `create_index` parameter is a boolean flag that specifies whether to
1790        create indexes on the exploded INFO fields. If set to `True`, indexes will be created; if set to
1791        `False`, indexes will not be created. The default value is `False`, defaults to False
1792        :type create_index: bool (optional)
1793        :param fields: The `fields` parameter in the `explode_infos` function is a list of INFO fields
1794        that you want to explode into individual columns. If this parameter is not provided, all INFO
1795        fields will be exploded. You can specify the INFO fields you want to explode by passing them as
1796        a list to the `
1797        :type fields: list
1798        :param force: The `force` parameter in the `explode_infos` function is a boolean flag that
1799        determines whether to drop and recreate a column if it already exists in the table. If `force`
1800        is set to `True`, the column will be dropped and recreated. If `force` is set to `False,
1801        defaults to False
1802        :type force: bool (optional)
1803        :param proccess_all_fields_together: The `proccess_all_fields_together` parameter is a boolean
1804        flag that determines whether to process all the INFO fields together or individually. If set to
1805        `True`, all the INFO fields will be processed together. If set to `False`, each INFO field will
1806        be processed individually. The default value is, defaults to False
1807        :type proccess_all_fields_together: bool (optional)
1808        :param table: The `table` parameter in the `explode_infos` function is used to specify the name
1809        of the table where the exploded INFO fields will be added as individual columns. If you provide
1810        a value for the `table` parameter, the function will use that table name. If the `table`
1811        parameter is
1812        :type table: str
1813        :return: The `explode_infos` function returns a list of added columns.
1814        """
1815
1816        # drop indexes
1817        self.drop_indexes()
1818
1819        # connexion format
1820        connexion_format = self.get_connexion_format()
1821
1822        # Access
1823        access = self.get_config().get("access", None)
1824
1825        # Added columns
1826        added_columns = []
1827
1828        if access not in ["RO"]:
1829
1830            # prefix
1831            if prefix in [None, True] or not isinstance(prefix, str):
1832                if self.get_explode_infos_prefix() not in [None, True]:
1833                    prefix = self.get_explode_infos_prefix()
1834                else:
1835                    prefix = "INFO/"
1836
1837            # table variants
1838            if table is not None:
1839                table_variants = table
1840            else:
1841                table_variants = self.get_table_variants(clause="select")
1842
1843            # extra infos
1844            try:
1845                extra_infos = self.get_extra_infos()
1846            except:
1847                extra_infos = []
1848
1849            # Header infos
1850            header_infos = self.get_header().infos
1851
1852            log.debug(
1853                f"Explode INFO fields - ADD [{len(header_infos)}] annotations fields"
1854            )
1855
1856            sql_info_alter_table_array = []
1857
1858            # Info fields to check
1859            fields_list = list(header_infos)
1860            if fields:
1861                fields_list += fields
1862            fields_list = set(fields_list)
1863
1864            # If no fields
1865            if not fields:
1866                fields = []
1867
1868            # Translate fields if patterns
1869            fields = self.get_explode_infos_fields(explode_infos_fields=fields)
1870
1871            for info in fields:
1872
1873                info_id_sql = prefix + info
1874
1875                if (
1876                    info in fields_list
1877                    or prefix + info in fields_list
1878                    or info in extra_infos
1879                ):
1880
1881                    log.debug(f"Explode INFO fields - ADD '{info}' annotations fields")
1882
1883                    if info in header_infos:
1884                        info_type = header_infos[info].type
1885                        info_num = header_infos[info].num
1886                    else:
1887                        info_type = "String"
1888                        info_num = 0
1889
1890                    type_sql = self.code_type_map_to_sql.get(info_type, "VARCHAR")
1891                    if info_num != 1:
1892                        type_sql = "VARCHAR"
1893
1894                    # Add field
1895                    added_column = self.add_column(
1896                        table_name=table_variants,
1897                        column_name=info_id_sql,
1898                        column_type=type_sql,
1899                        default_value="null",
1900                        drop=force,
1901                    )
1902
1903                    if added_column:
1904                        added_columns.append(added_column)
1905
1906                    if added_column or force:
1907
1908                        # add field to index
1909                        self.index_additionnal_fields.append(info_id_sql)
1910
1911                        # Update field array
1912                        if connexion_format in ["duckdb"]:
1913                            update_info_field = f"""
1914                            "{info_id_sql}" =
1915                                CASE
1916                                    WHEN REGEXP_EXTRACT(concat(';', INFO), ';{info}=([^;]*)',1) IN ('','.') THEN NULL
1917                                    ELSE REGEXP_EXTRACT(concat(';', INFO), ';{info}=([^;]*)',1)
1918                                END
1919                            """
1920                        elif connexion_format in ["sqlite"]:
1921                            update_info_field = f"""
1922                                "{info_id_sql}" =
1923                                    CASE
1924                                        WHEN instr(INFO, '{info}=') = 0 THEN NULL
1925                                        WHEN instr(substr(INFO, instr(INFO, '{info}=')+{len(info)+1}),';') = 0 THEN substr(substr(INFO, instr(INFO, '{info}=')+{len(info)+1}), instr(substr(INFO, instr(INFO, '{info}=')+{len(info)+1}), '=')+1)
1926                                        ELSE substr(substr(INFO, instr(INFO, '{info}=')+{len(info)+1}), instr(substr(INFO, instr(INFO, '{info}=')+{len(info)+1}), '=')+1, instr(substr(INFO, instr(INFO, '{info}=')+{len(info)+1}),';')-instr(substr(INFO, instr(INFO, '{info}=')+{len(info)+1}), '=')-1)
1927                                    END
1928                            """
1929
1930                        sql_info_alter_table_array.append(update_info_field)
1931
1932            if sql_info_alter_table_array:
1933
1934                # By chromosomes
1935                try:
1936                    chromosomes_list = list(
1937                        self.get_query_to_df(
1938                            f""" SELECT "#CHROM" FROM {table_variants} GROUP BY "#CHROM" """
1939                        )["#CHROM"]
1940                    )
1941                except:
1942                    chromosomes_list = [None]
1943
1944                for chrom in chromosomes_list:
1945                    log.debug(f"Explode INFO fields - Chromosome {chrom}...")
1946
1947                    # Where clause
1948                    where_clause = ""
1949                    if chrom and len(chromosomes_list) > 1:
1950                        where_clause = f""" WHERE "#CHROM" = '{chrom}' """
1951
1952                    # Update table
1953                    if proccess_all_fields_together:
1954                        sql_info_alter_table_array_join = ", ".join(
1955                            sql_info_alter_table_array
1956                        )
1957                        if sql_info_alter_table_array_join:
1958                            sql_info_alter_table = f"""
1959                                UPDATE {table_variants}
1960                                SET {sql_info_alter_table_array_join}
1961                                {where_clause}
1962                                """
1963                            log.debug(
1964                                f"Explode INFO fields - Explode all {len(sql_info_alter_table_array)} fields..."
1965                            )
1966                            # log.debug(sql_info_alter_table)
1967                            self.conn.execute(sql_info_alter_table)
1968                    else:
1969                        sql_info_alter_num = 0
1970                        for sql_info_alter in sql_info_alter_table_array:
1971                            sql_info_alter_num += 1
1972                            sql_info_alter_table = f"""
1973                                UPDATE {table_variants}
1974                                SET {sql_info_alter}
1975                                {where_clause}
1976                                """
1977                            log.debug(
1978                                f"Explode INFO fields - Explode field {sql_info_alter_num}/{len(sql_info_alter_table_array)}..."
1979                            )
1980                            # log.debug(sql_info_alter_table)
1981                            self.conn.execute(sql_info_alter_table)
1982
1983        # create indexes
1984        if create_index:
1985            self.create_indexes()
1986
1987        return added_columns

The explode_infos function in Python takes a VCF file and explodes the INFO fields into individual columns, returning a list of added columns.

Parameters

prefix: The prefix parameter is a string that is used as a prefix for the exploded INFO fields. If the prefix is not provided or is set to None, the function will use the value of self.get_explode_infos_prefix() as the prefix
create_index: The create_index parameter is a boolean flag that specifies whether to create indexes on the exploded INFO fields. If set to True, indexes will be created; if set to False, indexes will not be created. The default value is False, defaults to False
fields: The fields parameter in the explode_infos function is a list of INFO fields that you want to explode into individual columns. If this parameter is not provided, all INFO fields will be exploded. You can specify the INFO fields you want to explode by passing them as a list to the `
force: The force parameter in the explode_infos function is a boolean flag that determines whether to drop and recreate a column if it already exists in the table. If force is set to True, the column will be dropped and recreated. If force is set to `False, defaults to False
proccess_all_fields_together: The proccess_all_fields_together parameter is a boolean flag that determines whether to process all the INFO fields together or individually. If set to True, all the INFO fields will be processed together. If set to False, each INFO field will be processed individually. The default value is, defaults to False
table: The table parameter in the explode_infos function is used to specify the name of the table where the exploded INFO fields will be added as individual columns. If you provide a value for the table parameter, the function will use that table name. If the table parameter is

Returns

The explode_infos function returns a list of added columns.

def create_indexes(self) -> None: View Source

1989    def create_indexes(self) -> None:
1990        """
1991        Create indexes on the table after insertion
1992        """
1993
1994        # Access
1995        access = self.get_config().get("access", None)
1996
1997        # get table variants
1998        table_variants = self.get_table_variants("FROM")
1999
2000        if self.get_indexing() and access not in ["RO"]:
2001            # Create index
2002            sql_create_table_index = f'CREATE INDEX IF NOT EXISTS idx_{self.get_table_variants()} ON {table_variants} ("#CHROM", "POS", "REF", "ALT")'
2003            self.conn.execute(sql_create_table_index)
2004            sql_create_table_index = f'CREATE INDEX IF NOT EXISTS idx_{self.get_table_variants()}_chrom ON {table_variants} ("#CHROM")'
2005            self.conn.execute(sql_create_table_index)
2006            sql_create_table_index = f'CREATE INDEX IF NOT EXISTS idx_{self.get_table_variants()}_pos ON {table_variants} ("POS")'
2007            self.conn.execute(sql_create_table_index)
2008            sql_create_table_index = f'CREATE INDEX IF NOT EXISTS idx_{self.get_table_variants()}_ref ON {table_variants} ( "REF")'
2009            self.conn.execute(sql_create_table_index)
2010            sql_create_table_index = f'CREATE INDEX IF NOT EXISTS idx_{self.get_table_variants()}_alt ON {table_variants} ("ALT")'
2011            self.conn.execute(sql_create_table_index)
2012            for field in self.index_additionnal_fields:
2013                sql_create_table_index = f""" CREATE INDEX IF NOT EXISTS "idx_{self.get_table_variants()}_{field}" ON {table_variants} ("{field}") """
2014                self.conn.execute(sql_create_table_index)

Create indexes on the table after insertion

def drop_indexes(self) -> None: View Source

2016    def drop_indexes(self) -> None:
2017        """
2018        Create indexes on the table after insertion
2019        """
2020
2021        # Access
2022        access = self.get_config().get("access", None)
2023
2024        # get table variants
2025        table_variants = self.get_table_variants("FROM")
2026
2027        # Get database format
2028        connexion_format = self.get_connexion_format()
2029
2030        if access not in ["RO"]:
2031            if connexion_format in ["duckdb"]:
2032                sql_list_indexes = f"SELECT index_name FROM duckdb_indexes WHERE table_name='{table_variants}'"
2033            elif connexion_format in ["sqlite"]:
2034                sql_list_indexes = f"SELECT name FROM sqlite_master WHERE type='index' AND tbl_name='{table_variants}';"
2035
2036            list_indexes = self.conn.execute(sql_list_indexes)
2037            index_names = [row[0] for row in list_indexes.fetchall()]
2038            for index in index_names:
2039                sql_drop_table_index = f""" DROP INDEX IF EXISTS "{index}" """
2040                self.conn.execute(sql_drop_table_index)

Create indexes on the table after insertion

def read_vcf_header(self, f) -> list: View Source

2042    def read_vcf_header(self, f) -> list:
2043        """
2044        It reads the header of a VCF file and returns a list of the header lines
2045
2046        :param f: the file object
2047        :return: The header lines of the VCF file.
2048        """
2049
2050        header_list = []
2051        for line in f:
2052            header_list.append(line)
2053            if line.startswith("#CHROM"):
2054                break
2055        return header_list

It reads the header of a VCF file and returns a list of the header lines

Parameters

f: the file object

Returns

The header lines of the VCF file.

def read_vcf_header_file(self, file: str = None) -> list: View Source

2057    def read_vcf_header_file(self, file: str = None) -> list:
2058        """
2059        The `read_vcf_header_file` function reads the header of a VCF file, handling both compressed and
2060        uncompressed files.
2061
2062        :param file: The `file` parameter is a string that represents the path to the VCF header file
2063        that you want to read. It is an optional parameter, so if you don't provide a value, it will
2064        default to `None`
2065        :type file: str
2066        :return: The function `read_vcf_header_file` returns a list.
2067        """
2068
2069        if self.get_input_compressed(input_file=file):
2070            with bgzf.open(file, "rt") as f:
2071                return self.read_vcf_header(f=f)
2072        else:
2073            with open(file, "rt") as f:
2074                return self.read_vcf_header(f=f)

The read_vcf_header_file function reads the header of a VCF file, handling both compressed and uncompressed files.

Parameters

file: The file parameter is a string that represents the path to the VCF header file that you want to read. It is an optional parameter, so if you don't provide a value, it will default to None

Returns

The function read_vcf_header_file returns a list.

def execute_query(self, query: str): View Source

2076    def execute_query(self, query: str):
2077        """
2078        It takes a query as an argument, executes it, and returns the results
2079
2080        :param query: The query to be executed
2081        :return: The result of the query is being returned.
2082        """
2083        if query:
2084            return self.conn.execute(query)  # .fetchall()
2085        else:
2086            return None

It takes a query as an argument, executes it, and returns the results

Parameters

query: The query to be executed

Returns

The result of the query is being returned.

2088    def export_output(
2089        self,
2090        output_file: str | None = None,
2091        output_header: str | None = None,
2092        export_header: bool = True,
2093        query: str | None = None,
2094        parquet_partitions: list | None = None,
2095        chunk_size: int | None = None,
2096        threads: int | None = None,
2097        sort: bool = False,
2098        index: bool = False,
2099        order_by: str | None = None,
2100        fields_to_rename: dict | None = None,
2101    ) -> bool:
2102        """
2103        The `export_output` function exports data from a VCF file to various formats, including VCF,
2104        CSV, TSV, PSV, and Parquet, with options for customization such as filtering, sorting, and
2105        partitioning.
2106
2107        :param output_file: The `output_file` parameter is a string that specifies the name of the
2108        output file where the exported data will be saved
2109        :type output_file: str | None
2110        :param output_header: The `output_header` parameter is a string that specifies the name of the
2111        file where the header of the VCF file will be exported. If this parameter is not provided, the
2112        header will be exported to a file with the same name as the `output_file` parameter, but with
2113        the extension "
2114        :type output_header: str | None
2115        :param export_header: The `export_header` parameter is a boolean flag that determines whether
2116        the header of a VCF file should be exported to a separate file or not. If `export_header` is
2117        True, the header will be exported to a file. If `export_header` is False, the header will not
2118        be, defaults to True
2119        :type export_header: bool (optional)
2120        :param query: The `query` parameter in the `export_output` function is an optional SQL query
2121        that can be used to filter and select specific data from the VCF file before exporting it. If
2122        provided, only the data that matches the query will be exported. This allows you to customize
2123        the exported data based on
2124        :type query: str | None
2125        :param parquet_partitions: The `parquet_partitions` parameter is a list that specifies the
2126        columns to be used for partitioning the Parquet file during export. Partitioning is a way to
2127        organize data in a hierarchical directory structure based on the values of one or more columns.
2128        This can improve query performance when working with large datasets
2129        :type parquet_partitions: list | None
2130        :param chunk_size: The `chunk_size` parameter specifies the number of records in a batch when
2131        exporting data in Parquet format. This parameter is used for partitioning the Parquet file into
2132        multiple files. It helps in optimizing the export process by breaking down the data into
2133        manageable chunks for processing and storage
2134        :type chunk_size: int | None
2135        :param threads: The `threads` parameter in the `export_output` function specifies the number of
2136        threads to be used during the export process. It determines the level of parallelism and can
2137        improve the performance of the export operation. If this parameter is not provided, the function
2138        will use the default number of threads
2139        :type threads: int | None
2140        :param sort: The `sort` parameter in the `export_output` function is a boolean flag that
2141        determines whether the output file should be sorted based on genomic coordinates of the
2142        variants. If `sort` is set to `True`, the output file will be sorted. If `sort` is set to
2143        `False`,, defaults to False
2144        :type sort: bool (optional)
2145        :param index: The `index` parameter in the `export_output` function is a boolean flag that
2146        determines whether an index should be created on the output file. If `index` is set to `True`,
2147        an index will be created on the output file. If `index` is set to `False`, no, defaults to False
2148        :type index: bool (optional)
2149        :param order_by: The `order_by` parameter in the `export_output` function is a string that
2150        specifies the column(s) to use for sorting the output file. This parameter is only applicable
2151        when exporting data in VCF format. It allows you to specify the column(s) based on which the
2152        output file should be
2153        :type order_by: str | None
2154        :param fields_to_rename: The `fields_to_rename` parameter is a dictionary that specifies the
2155        mapping of field names to be renamed during the export process. This parameter allows you to
2156        customize the output field names before exporting the data. Each key-value pair in the
2157        dictionary represents the original field name as the key and the new field name
2158        :type fields_to_rename: dict | None
2159        :return: The `export_output` function returns a boolean value. It checks if the output file
2160        exists and returns True if it does, or None if it doesn't.
2161        """
2162
2163        # Log
2164        log.info("Exporting...")
2165
2166        # Full path
2167        output_file = full_path(output_file)
2168        output_header = full_path(output_header)
2169
2170        # Config
2171        config = self.get_config()
2172
2173        # Param
2174        param = self.get_param()
2175
2176        # Tmp files to remove
2177        tmp_to_remove = []
2178
2179        # If no output, get it
2180        if not output_file:
2181            output_file = self.get_output()
2182
2183        # If not threads
2184        if not threads:
2185            threads = self.get_threads()
2186
2187        # Rename fields
2188        if not fields_to_rename:
2189            fields_to_rename = param.get("export", {}).get("fields_to_rename", None)
2190        self.rename_info_fields(fields_to_rename=fields_to_rename)
2191
2192        # Auto header name with extension
2193        if export_header or output_header:
2194            if not output_header:
2195                output_header = f"{output_file}.hdr"
2196            # Export header
2197            self.export_header(output_file=output_file)
2198
2199        # Switch off export header if VCF output
2200        output_file_type = get_file_format(output_file)
2201        if output_file_type in ["vcf"]:
2202            export_header = False
2203            tmp_to_remove.append(output_header)
2204
2205        # Chunk size
2206        if not chunk_size:
2207            chunk_size = config.get("chunk_size", None)
2208
2209        # Parquet partition
2210        if not parquet_partitions:
2211            parquet_partitions = param.get("export", {}).get("parquet_partitions", None)
2212        if parquet_partitions and isinstance(parquet_partitions, str):
2213            parquet_partitions = parquet_partitions.split(",")
2214
2215        # Order by
2216        if not order_by:
2217            order_by = param.get("export", {}).get("order_by", "")
2218
2219        # Header in output
2220        header_in_output = param.get("export", {}).get("include_header", False)
2221
2222        # Database
2223        database_source = self.get_connexion()
2224
2225        # Connexion format
2226        connexion_format = self.get_connexion_format()
2227
2228        # Explode infos
2229        if self.get_explode_infos():
2230            self.explode_infos(
2231                prefix=self.get_explode_infos_prefix(),
2232                fields=self.get_explode_infos_fields(),
2233                force=False,
2234            )
2235
2236        # if connexion_format in ["sqlite"] or query:
2237        if connexion_format in ["sqlite"]:
2238
2239            # Export in Parquet
2240            random_tmp = "".join(
2241                random.choice(string.ascii_lowercase) for i in range(10)
2242            )
2243            database_source = f"""{output_file}.{random_tmp}.database_export.parquet"""
2244            tmp_to_remove.append(database_source)
2245
2246            # Table Variants
2247            table_variants = self.get_table_variants()
2248
2249            # Create export query
2250            sql_query_export_subquery = f"""
2251                SELECT * FROM {table_variants}
2252                """
2253
2254            # Write source file
2255            fp.write(database_source, self.get_query_to_df(sql_query_export_subquery))
2256
2257        # Create database
2258        database = Database(
2259            database=database_source,
2260            table="variants",
2261            header_file=output_header,
2262            conn_config=self.get_connexion_config(),
2263        )
2264
2265        # Existing colomns header
2266        existing_columns_header = database.get_header_columns_from_database(query=query)
2267
2268        # Sample list
2269        if output_file_type in ["vcf"]:
2270            get_samples = self.get_samples()
2271            get_samples_check = self.get_samples_check()
2272            samples_force = get_samples is not None
2273            sample_list = self.get_header_sample_list(
2274                check=get_samples_check,
2275                samples=get_samples,
2276                samples_force=samples_force,
2277            )
2278        else:
2279            sample_list = None
2280
2281        # Export file
2282        database.export(
2283            output_database=output_file,
2284            output_header=output_header,
2285            existing_columns_header=existing_columns_header,
2286            parquet_partitions=parquet_partitions,
2287            chunk_size=chunk_size,
2288            threads=threads,
2289            sort=sort,
2290            index=index,
2291            header_in_output=header_in_output,
2292            order_by=order_by,
2293            query=query,
2294            export_header=export_header,
2295            sample_list=sample_list,
2296        )
2297
2298        # Remove
2299        remove_if_exists(tmp_to_remove)
2300
2301        return (os.path.exists(output_file) or None) and (
2302            os.path.exists(output_file) or None
2303        )

The export_output function exports data from a VCF file to various formats, including VCF, CSV, TSV, PSV, and Parquet, with options for customization such as filtering, sorting, and partitioning.

Parameters

output_file: The output_file parameter is a string that specifies the name of the output file where the exported data will be saved
output_header: The output_header parameter is a string that specifies the name of the file where the header of the VCF file will be exported. If this parameter is not provided, the header will be exported to a file with the same name as the output_file parameter, but with the extension "
export_header: The export_header parameter is a boolean flag that determines whether the header of a VCF file should be exported to a separate file or not. If export_header is True, the header will be exported to a file. If export_header is False, the header will not be, defaults to True
query: The query parameter in the export_output function is an optional SQL query that can be used to filter and select specific data from the VCF file before exporting it. If provided, only the data that matches the query will be exported. This allows you to customize the exported data based on
parquet_partitions: The parquet_partitions parameter is a list that specifies the columns to be used for partitioning the Parquet file during export. Partitioning is a way to organize data in a hierarchical directory structure based on the values of one or more columns. This can improve query performance when working with large datasets
chunk_size: The chunk_size parameter specifies the number of records in a batch when exporting data in Parquet format. This parameter is used for partitioning the Parquet file into multiple files. It helps in optimizing the export process by breaking down the data into manageable chunks for processing and storage
threads: The threads parameter in the export_output function specifies the number of threads to be used during the export process. It determines the level of parallelism and can improve the performance of the export operation. If this parameter is not provided, the function will use the default number of threads
sort: The sort parameter in the export_output function is a boolean flag that determines whether the output file should be sorted based on genomic coordinates of the variants. If sort is set to True, the output file will be sorted. If sort is set to False,, defaults to False
index: The index parameter in the export_output function is a boolean flag that determines whether an index should be created on the output file. If index is set to True, an index will be created on the output file. If index is set to False, no, defaults to False
order_by: The order_by parameter in the export_output function is a string that specifies the column(s) to use for sorting the output file. This parameter is only applicable when exporting data in VCF format. It allows you to specify the column(s) based on which the output file should be
fields_to_rename: The fields_to_rename parameter is a dictionary that specifies the mapping of field names to be renamed during the export process. This parameter allows you to customize the output field names before exporting the data. Each key-value pair in the dictionary represents the original field name as the key and the new field name

Returns

The export_output function returns a boolean value. It checks if the output file exists and returns True if it does, or None if it doesn't.

def get_extra_infos(self, table: str = None) -> list: View Source

2305    def get_extra_infos(self, table: str = None) -> list:
2306        """
2307        The `get_extra_infos` function returns a list of columns that are in a specified table but not
2308        in the header.
2309
2310        :param table: The `table` parameter in the `get_extra_infos` function is used to specify the
2311        name of the table from which you want to retrieve the extra columns that are not present in the
2312        header. If the `table` parameter is not provided when calling the function, it will default to
2313        using the variants
2314        :type table: str
2315        :return: A list of columns that are in the specified table but not in the header of the table.
2316        """
2317
2318        header_columns = []
2319
2320        if not table:
2321            table = self.get_table_variants(clause="from")
2322            header_columns = self.get_header_columns()
2323
2324        # Check all columns in the database
2325        query = f""" SELECT * FROM {table} LIMIT 1 """
2326        log.debug(f"query {query}")
2327        table_columns = self.get_query_to_df(query).columns.tolist()
2328        extra_columns = []
2329
2330        # Construct extra infos (not in header)
2331        for column in table_columns:
2332            if column not in header_columns:
2333                extra_columns.append(column)
2334
2335        return extra_columns

The get_extra_infos function returns a list of columns that are in a specified table but not in the header.

Parameters

table: The table parameter in the get_extra_infos function is used to specify the name of the table from which you want to retrieve the extra columns that are not present in the header. If the table parameter is not provided when calling the function, it will default to using the variants

Returns

A list of columns that are in the specified table but not in the header of the table.

def get_extra_infos_sql(self, table: str = None) -> str: View Source

2337    def get_extra_infos_sql(self, table: str = None) -> str:
2338        """
2339        It returns a string of the extra infos, separated by commas, and each extra info is surrounded
2340        by double quotes
2341
2342        :param table: The name of the table to get the extra infos from. If None, the default table is
2343        used
2344        :type table: str
2345        :return: A string of the extra infos
2346        """
2347
2348        return ", ".join(
2349            ['"' + str(elem) + '"' for elem in self.get_extra_infos(table=table)]
2350        )

It returns a string of the extra infos, separated by commas, and each extra info is surrounded by double quotes

Parameters

table: The name of the table to get the extra infos from. If None, the default table is used

Returns

A string of the extra infos

def export_header( self, header_name: str = None, output_file: str = None, output_file_ext: str = '.hdr', clean_header: bool = True, remove_chrom_line: bool = False) -> str: View Source

2352    def export_header(
2353        self,
2354        header_name: str = None,
2355        output_file: str = None,
2356        output_file_ext: str = ".hdr",
2357        clean_header: bool = True,
2358        remove_chrom_line: bool = False,
2359    ) -> str:
2360        """
2361        The `export_header` function takes a VCF file, extracts the header, modifies it according to
2362        specified options, and writes it to a new file.
2363
2364        :param header_name: The `header_name` parameter is the name of the header file to be created. If
2365        this parameter is not specified, the header will be written to the output file
2366        :type header_name: str
2367        :param output_file: The `output_file` parameter in the `export_header` function is used to
2368        specify the name of the output file where the header will be written. If this parameter is not
2369        provided, the header will be written to a temporary file
2370        :type output_file: str
2371        :param output_file_ext: The `output_file_ext` parameter in the `export_header` function is a
2372        string that represents the extension of the output header file. By default, it is set to ".hdr"
2373        if not specified by the user. This extension will be appended to the `output_file` name to
2374        create the final, defaults to .hdr
2375        :type output_file_ext: str (optional)
2376        :param clean_header: The `clean_header` parameter in the `export_header` function is a boolean
2377        flag that determines whether the header should be cleaned or not. When `clean_header` is set to
2378        `True`, the function will clean the header by modifying certain lines based on a specific
2379        pattern. If `clean_header`, defaults to True
2380        :type clean_header: bool (optional)
2381        :param remove_chrom_line: The `remove_chrom_line` parameter in the `export_header` function is a
2382        boolean flag that determines whether the #CHROM line should be removed from the header before
2383        writing it to the output file. If set to `True`, the #CHROM line will be removed; if set to `,
2384        defaults to False
2385        :type remove_chrom_line: bool (optional)
2386        :return: The function `export_header` returns the name of the temporary header file that is
2387        created.
2388        """
2389
2390        if not header_name and not output_file:
2391            output_file = self.get_output()
2392
2393        if self.get_header():
2394
2395            # Get header object
2396            header_obj = self.get_header()
2397
2398            # Create database
2399            db_for_header = Database(database=self.get_input())
2400
2401            # Get real columns in the file
2402            db_header_columns = db_for_header.get_columns()
2403
2404            with tempfile.TemporaryDirectory() as tmpdir:
2405
2406                # Write header file
2407                header_file_tmp = os.path.join(tmpdir, "header")
2408                f = open(header_file_tmp, "w")
2409                vcf.Writer(f, header_obj)
2410                f.close()
2411
2412                # Replace #CHROM line with rel columns
2413                header_list = db_for_header.read_header_file(
2414                    header_file=header_file_tmp
2415                )
2416                header_list[-1] = "\t".join(db_header_columns)
2417
2418                # Remove CHROM line
2419                if remove_chrom_line:
2420                    header_list.pop()
2421
2422                # Clean header
2423                if clean_header:
2424                    header_list_clean = []
2425                    for head in header_list:
2426                        # Clean head for malformed header
2427                        head_clean = head
2428                        head_clean = re.subn(
2429                            "##FORMAT=<ID=(.*),Number=(.*),Type=Flag",
2430                            r"##FORMAT=<ID=\1,Number=\2,Type=String",
2431                            head_clean,
2432                            2,
2433                        )[0]
2434                        # Write header
2435                        header_list_clean.append(head_clean)
2436                    header_list = header_list_clean
2437
2438            tmp_header_name = output_file + output_file_ext
2439
2440            f = open(tmp_header_name, "w")
2441            for line in header_list:
2442                f.write(line)
2443            f.close()
2444
2445        return tmp_header_name

The export_header function takes a VCF file, extracts the header, modifies it according to specified options, and writes it to a new file.

Parameters

header_name: The header_name parameter is the name of the header file to be created. If this parameter is not specified, the header will be written to the output file
output_file: The output_file parameter in the export_header function is used to specify the name of the output file where the header will be written. If this parameter is not provided, the header will be written to a temporary file
output_file_ext: The output_file_ext parameter in the export_header function is a string that represents the extension of the output header file. By default, it is set to ".hdr" if not specified by the user. This extension will be appended to the output_file name to create the final, defaults to .hdr
clean_header: The clean_header parameter in the export_header function is a boolean flag that determines whether the header should be cleaned or not. When clean_header is set to True, the function will clean the header by modifying certain lines based on a specific pattern. If clean_header, defaults to True
remove_chrom_line: The remove_chrom_line parameter in the export_header function is a boolean flag that determines whether the #CHROM line should be removed from the header before writing it to the output file. If set to True, the #CHROM line will be removed; if set to `, defaults to False

Returns

The function export_header returns the name of the temporary header file that is created.

def export_variant_vcf( self, vcf_file, remove_info: bool = False, add_samples: bool = True, list_samples: list = [], where_clause: str = '', index: bool = False, threads: int | None = None) -> bool | None: View Source

2447    def export_variant_vcf(
2448        self,
2449        vcf_file,
2450        remove_info: bool = False,
2451        add_samples: bool = True,
2452        list_samples: list = [],
2453        where_clause: str = "",
2454        index: bool = False,
2455        threads: int | None = None,
2456    ) -> bool | None:
2457        """
2458        The `export_variant_vcf` function exports a VCF file with specified samples, allowing options to
2459        remove INFO field, add samples, and control compression and indexing.
2460
2461        :param vcf_file: The `vcf_file` parameter is the name of the file where the VCF data will be
2462        written to. It is the output file that will contain the filtered VCF data based on the specified
2463        parameters
2464        :param remove_info: The `remove_info` parameter in the `export_variant_vcf` function is a
2465        boolean flag that determines whether to remove the INFO field from the output VCF file. If set
2466        to `True`, the INFO field will be removed. If set to `False`, the INFO field will be included
2467        in, defaults to False
2468        :type remove_info: bool (optional)
2469        :param add_samples: The `add_samples` parameter is a boolean parameter that determines whether
2470        the samples should be added to the VCF file or not. If set to True, the samples will be added.
2471        If set to False, the samples will be removed. The default value is True, defaults to True
2472        :type add_samples: bool (optional)
2473        :param list_samples: The `list_samples` parameter is a list of samples that you want to include
2474        in the output VCF file. By default, all samples will be included. If you provide a list of
2475        samples, only those samples will be included in the output file
2476        :type list_samples: list
2477        :param index: The `index` parameter in the `export_variant_vcf` function is a boolean flag that
2478        determines whether or not to create an index for the output VCF file. If `index` is set to
2479        `True`, the output VCF file will be indexed using tabix. If `index`, defaults to False
2480        :type index: bool (optional)
2481        :param threads: The `threads` parameter in the `export_variant_vcf` function specifies the
2482        number of threads to use for exporting the VCF file. It determines how many parallel threads
2483        will be used during the export process. More threads can potentially speed up the export process
2484        by utilizing multiple cores of the processor. If
2485        :type threads: int | None
2486        :return: The `export_variant_vcf` function returns the result of calling the `export_output`
2487        method with various parameters including the output file, query, threads, sort flag, and index
2488        flag. The `export_output` method is responsible for exporting the VCF data based on the
2489        specified parameters and configurations provided in the `export_variant_vcf` function.
2490        """
2491
2492        # Config
2493        config = self.get_config()
2494
2495        # Extract VCF
2496        log.debug("Export VCF...")
2497
2498        # Table variants
2499        table_variants = self.get_table_variants()
2500
2501        # Threads
2502        if not threads:
2503            threads = self.get_threads()
2504
2505        # Info fields
2506        if remove_info:
2507            if not isinstance(remove_info, str):
2508                remove_info = "."
2509            info_field = f"""'{remove_info}' as INFO"""
2510        else:
2511            info_field = "INFO"
2512
2513        # Samples fields
2514        if add_samples:
2515            if not list_samples:
2516                list_samples = self.get_header_sample_list()
2517            if list_samples:
2518                samples_fields = " , FORMAT , " + " , ".join(
2519                    [f""" "{sample}" """ for sample in list_samples]
2520                )
2521            else:
2522                samples_fields = ""
2523            log.debug(f"samples_fields: {samples_fields}")
2524        else:
2525            samples_fields = ""
2526
2527        # Where clause
2528        if where_clause is None:
2529            where_clause = ""
2530
2531        # Variants
2532        select_fields = """ "#CHROM", POS, ID, REF, ALT, QUAL, FILTER """
2533        sql_query_select = f""" SELECT {select_fields}, {info_field} {samples_fields} FROM {table_variants} {where_clause} """
2534        log.debug(f"sql_query_select={sql_query_select}")
2535
2536        return self.export_output(
2537            output_file=vcf_file,
2538            output_header=None,
2539            export_header=True,
2540            query=sql_query_select,
2541            parquet_partitions=None,
2542            chunk_size=config.get("chunk_size", None),
2543            threads=threads,
2544            sort=True,
2545            index=index,
2546            order_by=None,
2547        )

The export_variant_vcf function exports a VCF file with specified samples, allowing options to remove INFO field, add samples, and control compression and indexing.

Parameters

vcf_file: The vcf_file parameter is the name of the file where the VCF data will be written to. It is the output file that will contain the filtered VCF data based on the specified parameters
remove_info: The remove_info parameter in the export_variant_vcf function is a boolean flag that determines whether to remove the INFO field from the output VCF file. If set to True, the INFO field will be removed. If set to False, the INFO field will be included in, defaults to False
add_samples: The add_samples parameter is a boolean parameter that determines whether the samples should be added to the VCF file or not. If set to True, the samples will be added. If set to False, the samples will be removed. The default value is True, defaults to True
list_samples: The list_samples parameter is a list of samples that you want to include in the output VCF file. By default, all samples will be included. If you provide a list of samples, only those samples will be included in the output file
index: The index parameter in the export_variant_vcf function is a boolean flag that determines whether or not to create an index for the output VCF file. If index is set to True, the output VCF file will be indexed using tabix. If index, defaults to False
threads: The threads parameter in the export_variant_vcf function specifies the number of threads to use for exporting the VCF file. It determines how many parallel threads will be used during the export process. More threads can potentially speed up the export process by utilizing multiple cores of the processor. If

Returns

The export_variant_vcf function returns the result of calling the export_output method with various parameters including the output file, query, threads, sort flag, and index flag. The export_output method is responsible for exporting the VCF data based on the specified parameters and configurations provided in the export_variant_vcf function.

def run_commands(self, commands: list = [], threads: int = 1) -> None: View Source

2549    def run_commands(self, commands: list = [], threads: int = 1) -> None:
2550        """
2551        It takes a list of commands and runs them in parallel using the number of threads specified
2552
2553        :param commands: A list of commands to run
2554        :param threads: The number of threads to use, defaults to 1 (optional)
2555        """
2556
2557        run_parallel_commands(commands, threads)

It takes a list of commands and runs them in parallel using the number of threads specified

Parameters

commands: A list of commands to run
threads: The number of threads to use, defaults to 1 (optional)

def get_threads(self, default: int = 1) -> int: View Source

2559    def get_threads(self, default: int = 1) -> int:
2560        """
2561        This function returns the number of threads to use for a job, with a default value of 1 if not
2562        specified.
2563
2564        :param default: The `default` parameter in the `get_threads` method is used to specify the
2565        default number of threads to use if no specific value is provided. If no value is provided for
2566        the `threads` parameter in the configuration or input parameters, the `default` value will be
2567        used, defaults to 1
2568        :type default: int (optional)
2569        :return: the number of threads to use for the current job.
2570        """
2571
2572        # Config
2573        config = self.get_config()
2574
2575        # Param
2576        param = self.get_param()
2577
2578        # Input threads
2579        input_thread = param.get("threads", config.get("threads", None))
2580
2581        # Check threads
2582        if not input_thread:
2583            threads = default
2584        elif int(input_thread) <= 0:
2585            threads = os.cpu_count()
2586        else:
2587            threads = int(input_thread)
2588        return threads

This function returns the number of threads to use for a job, with a default value of 1 if not specified.

Parameters

default: The default parameter in the get_threads method is used to specify the default number of threads to use if no specific value is provided. If no value is provided for the threads parameter in the configuration or input parameters, the default value will be used, defaults to 1

Returns

the number of threads to use for the current job.

def get_memory(self, default: str = None) -> str: View Source

2590    def get_memory(self, default: str = None) -> str:
2591        """
2592        This function retrieves the memory value from parameters or configuration with a default value
2593        if not found.
2594
2595        :param default: The `get_memory` function takes in a default value as a string parameter. This
2596        default value is used as a fallback in case the `memory` parameter is not provided in the
2597        `param` dictionary or the `config` dictionary. If `memory` is not found in either dictionary,
2598        the function
2599        :type default: str
2600        :return: The `get_memory` function returns a string value representing the memory parameter. If
2601        the `input_memory` is provided in the parameters, it will return that value. Otherwise, it will
2602        return the default value provided as an argument to the function.
2603        """
2604
2605        # Config
2606        config = self.get_config()
2607
2608        # Param
2609        param = self.get_param()
2610
2611        # Input threads
2612        input_memory = param.get("memory", config.get("memory", None))
2613
2614        # Check threads
2615        if input_memory:
2616            memory = input_memory
2617        else:
2618            memory = default
2619
2620        return memory

This function retrieves the memory value from parameters or configuration with a default value if not found.

Parameters

default: The get_memory function takes in a default value as a string parameter. This default value is used as a fallback in case the memory parameter is not provided in the param dictionary or the config dictionary. If memory is not found in either dictionary, the function

Returns

The get_memory function returns a string value representing the memory parameter. If the input_memory is provided in the parameters, it will return that value. Otherwise, it will return the default value provided as an argument to the function.

def update_from_vcf(self, vcf_file: str) -> None: View Source

2622    def update_from_vcf(self, vcf_file: str) -> None:
2623        """
2624        > If the database is duckdb, then use the parquet method, otherwise use the sqlite method
2625
2626        :param vcf_file: the path to the VCF file
2627        """
2628
2629        connexion_format = self.get_connexion_format()
2630
2631        if connexion_format in ["duckdb"]:
2632            self.update_from_vcf_duckdb(vcf_file)
2633        elif connexion_format in ["sqlite"]:
2634            self.update_from_vcf_sqlite(vcf_file)

If the database is duckdb, then use the parquet method, otherwise use the sqlite method

Parameters

vcf_file: the path to the VCF file

def update_from_vcf_duckdb(self, vcf_file: str) -> None: View Source

2636    def update_from_vcf_duckdb(self, vcf_file: str) -> None:
2637        """
2638        It takes a VCF file and updates the INFO column of the variants table in the database with the
2639        INFO column of the VCF file
2640
2641        :param vcf_file: the path to the VCF file
2642        """
2643
2644        # varaints table
2645        table_variants = self.get_table_variants()
2646
2647        # Loading VCF into temporaire table
2648        skip = self.get_header_length(file=vcf_file)
2649        vcf_df = pd.read_csv(
2650            vcf_file,
2651            sep="\t",
2652            engine="c",
2653            skiprows=skip,
2654            header=0,
2655            low_memory=False,
2656        )
2657        sql_query_update = f"""
2658        UPDATE {table_variants} as table_variants
2659            SET INFO = concat(
2660                            CASE
2661                                WHEN INFO NOT IN ('', '.')
2662                                THEN INFO
2663                                ELSE ''
2664                            END,
2665                            (
2666                                SELECT 
2667                                    concat(
2668                                        CASE
2669                                            WHEN table_variants.INFO NOT IN ('','.') AND table_parquet.INFO NOT IN ('','.')
2670                                            THEN ';'
2671                                            ELSE ''
2672                                        END
2673                                        ,
2674                                        CASE
2675                                            WHEN table_parquet.INFO NOT IN ('','.')
2676                                            THEN table_parquet.INFO
2677                                            ELSE ''
2678                                        END
2679                                    )
2680                                FROM vcf_df as table_parquet
2681                                        WHERE CAST(table_parquet.\"#CHROM\" AS VARCHAR) = CAST(table_variants.\"#CHROM\" AS VARCHAR)
2682                                        AND table_parquet.\"POS\" = table_variants.\"POS\"
2683                                        AND table_parquet.\"ALT\" = table_variants.\"ALT\"
2684                                        AND table_parquet.\"REF\" = table_variants.\"REF\"
2685                                        AND table_parquet.INFO NOT IN ('','.')
2686                            )
2687                        )
2688            ;
2689            """
2690        self.conn.execute(sql_query_update)

It takes a VCF file and updates the INFO column of the variants table in the database with the INFO column of the VCF file

Parameters

vcf_file: the path to the VCF file

def update_from_vcf_sqlite(self, vcf_file: str) -> None: View Source

2692    def update_from_vcf_sqlite(self, vcf_file: str) -> None:
2693        """
2694        It creates a temporary table in the SQLite database, loads the VCF file into the temporary
2695        table, then updates the INFO column of the variants table with the INFO column of the temporary
2696        table
2697
2698        :param vcf_file: The path to the VCF file you want to update the database with
2699        """
2700
2701        # Create a temporary table for the VCF
2702        table_vcf = "tmp_vcf"
2703        sql_create = (
2704            f"CREATE TEMPORARY TABLE {table_vcf} AS SELECT * FROM variants WHERE 0"
2705        )
2706        self.conn.execute(sql_create)
2707
2708        # Loading VCF into temporaire table
2709        vcf_df = pd.read_csv(
2710            vcf_file, sep="\t", comment="#", header=None, low_memory=False
2711        )
2712        vcf_df.columns = ["#CHROM", "POS", "ID", "REF", "ALT", "QUAL", "FILTER", "INFO"]
2713        vcf_df.to_sql(table_vcf, self.conn, if_exists="append", index=False)
2714
2715        # Update table 'variants' with VCF data
2716        # warning: CONCAT as || operator
2717        sql_query_update = f"""
2718            UPDATE variants as table_variants
2719            SET INFO = CASE
2720                            WHEN INFO NOT IN ('', '.')
2721                            THEN INFO
2722                            ELSE ''
2723                        END ||
2724                        (
2725                        SELECT 
2726                            CASE 
2727                                WHEN table_variants.INFO NOT IN ('','.') 
2728                                    AND table_vcf.INFO NOT IN ('','.')  
2729                                THEN ';' 
2730                                ELSE '' 
2731                            END || 
2732                            CASE 
2733                                WHEN table_vcf.INFO NOT IN ('','.') 
2734                                THEN table_vcf.INFO 
2735                                ELSE '' 
2736                            END
2737                        FROM {table_vcf} as table_vcf
2738                        WHERE table_vcf.\"#CHROM\" = table_variants.\"#CHROM\"
2739                            AND table_vcf.\"POS\" = table_variants.\"POS\"
2740                            AND table_vcf.\"ALT\" = table_variants.\"ALT\"
2741                            AND table_vcf.\"REF\" = table_variants.\"REF\"
2742                        )
2743        """
2744        self.conn.execute(sql_query_update)
2745
2746        # Drop temporary table
2747        sql_drop = f"DROP TABLE {table_vcf}"
2748        self.conn.execute(sql_drop)

It creates a temporary table in the SQLite database, loads the VCF file into the temporary table, then updates the INFO column of the variants table with the INFO column of the temporary table

Parameters

vcf_file: The path to the VCF file you want to update the database with

def drop_variants_table(self) -> None: View Source

2750    def drop_variants_table(self) -> None:
2751        """
2752        > This function drops the variants table
2753        """
2754
2755        table_variants = self.get_table_variants()
2756        sql_table_variants = f"DROP TABLE IF EXISTS {table_variants}"
2757        self.conn.execute(sql_table_variants)

This function drops the variants table

def set_variant_id(self, variant_id_column: str = 'variant_id', force: bool = None) -> str: View Source

2759    def set_variant_id(
2760        self, variant_id_column: str = "variant_id", force: bool = None
2761    ) -> str:
2762        """
2763        It adds a column to the variants table called `variant_id` and populates it with a hash of the
2764        `#CHROM`, `POS`, `REF`, and `ALT` columns
2765
2766        :param variant_id_column: The name of the column to be created in the variants table, defaults
2767        to variant_id
2768        :type variant_id_column: str (optional)
2769        :param force: If True, the variant_id column will be created even if it already exists
2770        :type force: bool
2771        :return: The name of the column that contains the variant_id
2772        """
2773
2774        # Assembly
2775        assembly = self.get_param().get(
2776            "assembly", self.get_config().get("assembly", DEFAULT_ASSEMBLY)
2777        )
2778
2779        # INFO/Tag prefix
2780        prefix = self.get_explode_infos_prefix()
2781
2782        # Explode INFO/SVTYPE
2783        added_columns = self.explode_infos(prefix=prefix, fields=["SVTYPE"])
2784
2785        # variants table
2786        table_variants = self.get_table_variants()
2787
2788        # variant_id column
2789        if not variant_id_column:
2790            variant_id_column = "variant_id"
2791
2792        # Creta variant_id column
2793        if "variant_id" not in self.get_extra_infos() or force:
2794
2795            # Create column
2796            self.add_column(
2797                table_name=table_variants,
2798                column_name=variant_id_column,
2799                column_type="UBIGINT",
2800                default_value="0",
2801            )
2802
2803            # Update column
2804            self.conn.execute(
2805                f"""
2806                    UPDATE {table_variants}
2807                    SET "{variant_id_column}" = hash('{assembly}', "#CHROM", "POS", "REF", "ALT", '"{prefix}SVTYPE"')
2808                """
2809            )
2810
2811        # Remove added columns
2812        for added_column in added_columns:
2813            self.drop_column(column=added_column)
2814
2815        # return variant_id column name
2816        return variant_id_column

It adds a column to the variants table called variant_id and populates it with a hash of the #CHROM, POS, REF, and ALT columns

Parameters

variant_id_column: The name of the column to be created in the variants table, defaults to variant_id
force: If True, the variant_id column will be created even if it already exists

Returns

The name of the column that contains the variant_id

def get_variant_id_column(self, variant_id_column: str = 'variant_id', force: bool = None) -> str: View Source

2818    def get_variant_id_column(
2819        self, variant_id_column: str = "variant_id", force: bool = None
2820    ) -> str:
2821        """
2822        This function returns the variant_id column name
2823
2824        :param variant_id_column: The name of the column in the dataframe that contains the variant IDs,
2825        defaults to variant_id
2826        :type variant_id_column: str (optional)
2827        :param force: If True, will force the variant_id to be set to the value of variant_id_column. If
2828        False, will only set the variant_id if it is not already set. If None, will set the variant_id
2829        if it is not already set, or if it is set
2830        :type force: bool
2831        :return: The variant_id column name.
2832        """
2833
2834        return self.set_variant_id(variant_id_column=variant_id_column, force=force)

This function returns the variant_id column name

Parameters

variant_id_column: The name of the column in the dataframe that contains the variant IDs, defaults to variant_id
force: If True, will force the variant_id to be set to the value of variant_id_column. If False, will only set the variant_id if it is not already set. If None, will set the variant_id if it is not already set, or if it is set

Returns

The variant_id column name.

def scan_databases( self, database_formats: list = ['parquet'], database_releases: list = ['current']) -> dict: View Source

2840    def scan_databases(
2841        self,
2842        database_formats: list = ["parquet"],
2843        database_releases: list = ["current"],
2844    ) -> dict:
2845        """
2846        The function `scan_databases` scans for available databases based on specified formats and
2847        releases.
2848
2849        :param database_formats: The `database_formats` parameter is a list that specifies the formats
2850        of the databases to be scanned. In this case, the accepted format is "parquet"
2851        :type database_formats: list ["parquet"]
2852        :param database_releases: The `database_releases` parameter is a list that specifies the
2853        releases of the databases to be scanned. In the provided function, the default value for
2854        `database_releases` is set to `["current"]`, meaning that by default, the function will scan
2855        databases that are in the "current"
2856        :type database_releases: list
2857        :return: The function `scan_databases` returns a dictionary containing information about
2858        databases that match the specified formats and releases.
2859        """
2860
2861        # Config
2862        config = self.get_config()
2863
2864        # Param
2865        param = self.get_param()
2866
2867        # Param - Assembly
2868        assembly = param.get("assembly", config.get("assembly", None))
2869        if not assembly:
2870            assembly = DEFAULT_ASSEMBLY
2871            log.warning(f"Default assembly '{assembly}'")
2872
2873        # Scan for availabled databases
2874        log.info(
2875            f"Annotations - Check annotation parameters - Scan existing databases - Assembly {[assembly]} - Formats {database_formats} - Releases {database_releases}..."
2876        )
2877        databases_infos_dict = databases_infos(
2878            database_folder_releases=database_releases,
2879            database_formats=database_formats,
2880            assembly=assembly,
2881            config=config,
2882        )
2883        log.info(
2884            f"Annotations - Check annotation parameters - Scan existing databases - {len(databases_infos_dict)} databases found"
2885        )
2886
2887        return databases_infos_dict

The function scan_databases scans for available databases based on specified formats and releases.

Parameters

database_formats: The database_formats parameter is a list that specifies the formats of the databases to be scanned. In this case, the accepted format is "parquet"
database_releases: The database_releases parameter is a list that specifies the releases of the databases to be scanned. In the provided function, the default value for database_releases is set to ["current"], meaning that by default, the function will scan databases that are in the "current"

Returns

The function scan_databases returns a dictionary containing information about databases that match the specified formats and releases.

def annotation(self) -> None: View Source

2889    def annotation(self) -> None:
2890        """
2891        It annotates the VCF file with the annotations specified in the config file.
2892        """
2893
2894        # Config
2895        config = self.get_config()
2896
2897        # Param
2898        param = self.get_param()
2899
2900        # Param - Assembly
2901        assembly = param.get("assembly", config.get("assembly", None))
2902        if not assembly:
2903            assembly = DEFAULT_ASSEMBLY
2904            log.warning(f"Default assembly '{assembly}'")
2905
2906        # annotations databases folders
2907        annotations_databases = set(
2908            config.get("folders", {})
2909            .get("databases", {})
2910            .get("annotations", [DEFAULT_ANNOTATIONS_FOLDER])
2911            + config.get("folders", {})
2912            .get("databases", {})
2913            .get("parquet", ["~/howard/databases/parquet/current"])
2914            + config.get("folders", {})
2915            .get("databases", {})
2916            .get("bcftools", ["~/howard/databases/bcftools/current"])
2917        )
2918
2919        # Get param annotations
2920        if param.get("annotations", None) and isinstance(
2921            param.get("annotations", None), str
2922        ):
2923            log.debug(param.get("annotations", None))
2924            param_annotation_list = param.get("annotations").split(",")
2925        else:
2926            param_annotation_list = []
2927
2928        # Each tools param
2929        if param.get("annotation_parquet", None) != None:
2930            log.debug(
2931                f"""param.get("annotation_parquet", None)={param.get("annotation_parquet", None)}"""
2932            )
2933            if isinstance(param.get("annotation_parquet", None), list):
2934                param_annotation_list.append(",".join(param.get("annotation_parquet")))
2935            else:
2936                param_annotation_list.append(param.get("annotation_parquet"))
2937        if param.get("annotation_snpsift", None) != None:
2938            if isinstance(param.get("annotation_snpsift", None), list):
2939                param_annotation_list.append(
2940                    "snpsift:"
2941                    + "+".join(param.get("annotation_snpsift")).replace(",", "+")
2942                )
2943            else:
2944                param_annotation_list.append(
2945                    "snpsift:" + param.get("annotation_snpsift").replace(",", "+")
2946                )
2947        if param.get("annotation_snpeff", None) != None:
2948            param_annotation_list.append("snpeff:" + param.get("annotation_snpeff"))
2949        if param.get("annotation_bcftools", None) != None:
2950            if isinstance(param.get("annotation_bcftools", None), list):
2951                param_annotation_list.append(
2952                    "bcftools:"
2953                    + "+".join(param.get("annotation_bcftools")).replace(",", "+")
2954                )
2955            else:
2956                param_annotation_list.append(
2957                    "bcftools:" + param.get("annotation_bcftools").replace(",", "+")
2958                )
2959        if param.get("annotation_annovar", None) != None:
2960            param_annotation_list.append("annovar:" + param.get("annotation_annovar"))
2961        if param.get("annotation_exomiser", None) != None:
2962            param_annotation_list.append("exomiser:" + param.get("annotation_exomiser"))
2963        if param.get("annotation_splice", None) != None:
2964            param_annotation_list.append("splice:" + param.get("annotation_splice"))
2965
2966        # Merge param annotations list
2967        param["annotations"] = ",".join(param_annotation_list)
2968
2969        # debug
2970        log.debug(f"param_annotations={param['annotations']}")
2971
2972        if param.get("annotations"):
2973
2974            # Log
2975            # log.info("Annotations - Check annotation parameters")
2976
2977            if not "annotation" in param:
2978                param["annotation"] = {}
2979
2980            # List of annotations parameters
2981            annotations_list_input = {}
2982            if isinstance(param.get("annotations", None), str):
2983                annotation_file_list = [
2984                    value for value in param.get("annotations", "").split(",")
2985                ]
2986                for annotation_file in annotation_file_list:
2987                    annotations_list_input[annotation_file.strip()] = {"INFO": None}
2988            else:
2989                annotations_list_input = param.get("annotations", {})
2990
2991            log.info(f"Quick Annotations:")
2992            for annotation_key in list(annotations_list_input.keys()):
2993                log.info(f"   {annotation_key}")
2994
2995            # List of annotations and associated fields
2996            annotations_list = {}
2997
2998            for annotation_file in annotations_list_input:
2999
3000                # Explode annotations if ALL
3001                if (
3002                    annotation_file.upper() == "ALL"
3003                    or annotation_file.upper().startswith("ALL:")
3004                ):
3005
3006                    # check ALL parameters (formats, releases)
3007                    annotation_file_split = annotation_file.split(":")
3008                    database_formats = "parquet"
3009                    database_releases = "current"
3010                    for annotation_file_option in annotation_file_split[1:]:
3011                        database_all_options_split = annotation_file_option.split("=")
3012                        if database_all_options_split[0] == "format":
3013                            database_formats = database_all_options_split[1].split("+")
3014                        if database_all_options_split[0] == "release":
3015                            database_releases = database_all_options_split[1].split("+")
3016
3017                    # Scan for availabled databases
3018                    databases_infos_dict = self.scan_databases(
3019                        database_formats=database_formats,
3020                        database_releases=database_releases,
3021                    )
3022
3023                    # Add found databases in annotation parameters
3024                    for database_infos in databases_infos_dict.keys():
3025                        annotations_list[database_infos] = {"INFO": None}
3026
3027                else:
3028                    annotations_list[annotation_file] = annotations_list_input[
3029                        annotation_file
3030                    ]
3031
3032            # Check each databases
3033            if len(annotations_list):
3034
3035                log.info(
3036                    f"Annotations - Check annotation parameters - Check {len(annotations_list)} databases..."
3037                )
3038
3039                for annotation_file in annotations_list:
3040
3041                    # Init
3042                    annotations = annotations_list.get(annotation_file, None)
3043
3044                    # Annotation snpEff
3045                    if annotation_file.startswith("snpeff"):
3046
3047                        log.debug(f"Quick Annotation snpEff")
3048
3049                        if "snpeff" not in param["annotation"]:
3050                            param["annotation"]["snpeff"] = {}
3051
3052                        if "options" not in param["annotation"]["snpeff"]:
3053                            param["annotation"]["snpeff"]["options"] = ""
3054
3055                        # snpEff options in annotations
3056                        param["annotation"]["snpeff"]["options"] = "".join(
3057                            annotation_file.split(":")[1:]
3058                        )
3059
3060                    # Annotation Annovar
3061                    elif annotation_file.startswith("annovar"):
3062
3063                        log.debug(f"Quick Annotation Annovar")
3064
3065                        if "annovar" not in param["annotation"]:
3066                            param["annotation"]["annovar"] = {}
3067
3068                        if "annotations" not in param["annotation"]["annovar"]:
3069                            param["annotation"]["annovar"]["annotations"] = {}
3070
3071                        # Options
3072                        annotation_file_split = annotation_file.split(":")
3073                        for annotation_file_annotation in annotation_file_split[1:]:
3074                            if annotation_file_annotation:
3075                                param["annotation"]["annovar"]["annotations"][
3076                                    annotation_file_annotation
3077                                ] = annotations
3078
3079                    # Annotation Exomiser
3080                    elif annotation_file.startswith("exomiser"):
3081
3082                        log.debug(f"Quick Annotation Exomiser")
3083
3084                        param["annotation"]["exomiser"] = params_string_to_dict(
3085                            annotation_file
3086                        )
3087
3088                    # Annotation Splice
3089                    elif annotation_file.startswith("splice"):
3090
3091                        log.debug(f"Quick Annotation Splice")
3092
3093                        param["annotation"]["splice"] = params_string_to_dict(
3094                            annotation_file
3095                        )
3096
3097                    # Annotation Parquet or BCFTOOLS
3098                    else:
3099
3100                        # Tools detection
3101                        if annotation_file.startswith("bcftools:"):
3102                            annotation_tool_initial = "bcftools"
3103                            annotation_file = ":".join(annotation_file.split(":")[1:])
3104                        elif annotation_file.startswith("snpsift:"):
3105                            annotation_tool_initial = "snpsift"
3106                            annotation_file = ":".join(annotation_file.split(":")[1:])
3107                        elif annotation_file.startswith("bigwig:"):
3108                            annotation_tool_initial = "bigwig"
3109                            annotation_file = ":".join(annotation_file.split(":")[1:])
3110                        else:
3111                            annotation_tool_initial = None
3112
3113                        # list of files
3114                        annotation_file_list = annotation_file.replace("+", ":").split(
3115                            ":"
3116                        )
3117
3118                        for annotation_file in annotation_file_list:
3119
3120                            if annotation_file:
3121
3122                                # Annotation tool initial
3123                                annotation_tool = annotation_tool_initial
3124
3125                                # Find file
3126                                annotation_file_found = None
3127
3128                                if os.path.exists(annotation_file):
3129                                    annotation_file_found = annotation_file
3130                                elif os.path.exists(full_path(annotation_file)):
3131                                    annotation_file_found = full_path(annotation_file)
3132                                else:
3133                                    # Find within assembly folders
3134                                    for annotations_database in annotations_databases:
3135                                        found_files = find_all(
3136                                            annotation_file,
3137                                            os.path.join(
3138                                                annotations_database, assembly
3139                                            ),
3140                                        )
3141                                        if len(found_files) > 0:
3142                                            annotation_file_found = found_files[0]
3143                                            break
3144                                    if not annotation_file_found and not assembly:
3145                                        # Find within folders
3146                                        for (
3147                                            annotations_database
3148                                        ) in annotations_databases:
3149                                            found_files = find_all(
3150                                                annotation_file, annotations_database
3151                                            )
3152                                            if len(found_files) > 0:
3153                                                annotation_file_found = found_files[0]
3154                                                break
3155                                log.debug(
3156                                    f"for {annotation_file} annotation_file_found={annotation_file_found}"
3157                                )
3158
3159                                # Full path
3160                                annotation_file_found = full_path(annotation_file_found)
3161
3162                                if annotation_file_found:
3163
3164                                    database = Database(database=annotation_file_found)
3165                                    quick_annotation_format = database.get_format()
3166                                    quick_annotation_is_compressed = (
3167                                        database.is_compressed()
3168                                    )
3169                                    quick_annotation_is_indexed = os.path.exists(
3170                                        f"{annotation_file_found}.tbi"
3171                                    )
3172                                    bcftools_preference = False
3173
3174                                    # Check Annotation Tool
3175                                    if not annotation_tool:
3176                                        if (
3177                                            bcftools_preference
3178                                            and quick_annotation_format
3179                                            in ["vcf", "bed"]
3180                                            and quick_annotation_is_compressed
3181                                            and quick_annotation_is_indexed
3182                                        ):
3183                                            annotation_tool = "bcftools"
3184                                        elif quick_annotation_format in [
3185                                            "vcf",
3186                                            "bed",
3187                                            "tsv",
3188                                            "tsv",
3189                                            "csv",
3190                                            "json",
3191                                            "tbl",
3192                                            "parquet",
3193                                            "duckdb",
3194                                        ]:
3195                                            annotation_tool = "parquet"
3196                                        elif quick_annotation_format in ["bw"]:
3197                                            annotation_tool = "bigwig"
3198                                        else:
3199                                            log.error(
3200                                                f"Quick Annotation File {annotation_file_found} - Format {quick_annotation_format} not supported yet"
3201                                            )
3202                                            raise ValueError(
3203                                                f"Quick Annotation File {annotation_file_found} - Format {quick_annotation_format} not supported yet"
3204                                            )
3205
3206                                    log.debug(
3207                                        f"Quick Annotation File {annotation_file} - Annotation tool: {annotation_tool}"
3208                                    )
3209
3210                                    # Annotation Tool dispatch
3211                                    if annotation_tool:
3212                                        if annotation_tool not in param["annotation"]:
3213                                            param["annotation"][annotation_tool] = {}
3214                                        if (
3215                                            "annotations"
3216                                            not in param["annotation"][annotation_tool]
3217                                        ):
3218                                            param["annotation"][annotation_tool][
3219                                                "annotations"
3220                                            ] = {}
3221                                        param["annotation"][annotation_tool][
3222                                            "annotations"
3223                                        ][annotation_file_found] = annotations
3224
3225                                else:
3226                                    log.warning(
3227                                        f"Quick Annotation File {annotation_file} does NOT exist"
3228                                    )
3229
3230                self.set_param(param)
3231
3232        if param.get("annotation", None):
3233            log.info("Annotations")
3234            if param.get("annotation", {}).get("parquet", None):
3235                log.info("Annotations 'parquet'...")
3236                self.annotation_parquet()
3237            if param.get("annotation", {}).get("bcftools", None):
3238                log.info("Annotations 'bcftools'...")
3239                self.annotation_bcftools()
3240            if param.get("annotation", {}).get("snpsift", None):
3241                log.info("Annotations 'snpsift'...")
3242                self.annotation_snpsift()
3243            if param.get("annotation", {}).get("bigwig", None):
3244                log.info("Annotations 'bigwig'...")
3245                self.annotation_bigwig()
3246            if param.get("annotation", {}).get("annovar", None):
3247                log.info("Annotations 'annovar'...")
3248                self.annotation_annovar()
3249            if param.get("annotation", {}).get("snpeff", None):
3250                log.info("Annotations 'snpeff'...")
3251                self.annotation_snpeff()
3252            if param.get("annotation", {}).get("exomiser", None) is not None:
3253                log.info("Annotations 'exomiser'...")
3254                self.annotation_exomiser()
3255            if param.get("annotation", {}).get("splice", None) is not None:
3256                log.info("Annotations 'splice' ...")
3257                self.annotation_splice()
3258
3259        # Explode INFOS fields into table fields
3260        if self.get_explode_infos():
3261            self.explode_infos(
3262                prefix=self.get_explode_infos_prefix(),
3263                fields=self.get_explode_infos_fields(),
3264                force=True,
3265            )

It annotates the VCF file with the annotations specified in the config file.

def annotation_bigwig(self, threads: int = None) -> None: View Source

3267    def annotation_bigwig(self, threads: int = None) -> None:
3268        """
3269        The function `annotation_bigwig` annotates variants in a VCF file using bigwig databases.
3270
3271        :param threads: The `threads` parameter in the `annotation_bigwig` method is used to specify the
3272        number of threads to be used for parallel processing during the annotation process. If the
3273        `threads` parameter is not provided, the method will attempt to determine the optimal number of
3274        threads to use based on the system configuration
3275        :type threads: int
3276        :return: True
3277        """
3278
3279        # DEBUG
3280        log.debug("Start annotation with bigwig databases")
3281
3282        # # Threads
3283        # if not threads:
3284        #     threads = self.get_threads()
3285        # log.debug("Threads: " + str(threads))
3286
3287        # Config
3288        config = self.get_config()
3289        log.debug("Config: " + str(config))
3290
3291        # Config - BCFTools databases folders
3292        databases_folders = set(
3293            self.get_config()
3294            .get("folders", {})
3295            .get("databases", {})
3296            .get("annotations", ["."])
3297            + self.get_config()
3298            .get("folders", {})
3299            .get("databases", {})
3300            .get("bigwig", ["."])
3301        )
3302        log.debug("Databases annotations: " + str(databases_folders))
3303
3304        # Param
3305        annotations = (
3306            self.get_param()
3307            .get("annotation", {})
3308            .get("bigwig", {})
3309            .get("annotations", None)
3310        )
3311        log.debug("Annotations: " + str(annotations))
3312
3313        # Assembly
3314        assembly = self.get_param().get(
3315            "assembly", self.get_config().get("assembly", DEFAULT_ASSEMBLY)
3316        )
3317
3318        # Data
3319        table_variants = self.get_table_variants()
3320
3321        # Check if not empty
3322        log.debug("Check if not empty")
3323        sql_query_chromosomes = (
3324            f"""SELECT count(*) as count FROM {table_variants} as table_variants"""
3325        )
3326        sql_query_chromosomes_df = self.get_query_to_df(sql_query_chromosomes)
3327        if not sql_query_chromosomes_df["count"][0]:
3328            log.info(f"VCF empty")
3329            return
3330
3331        # VCF header
3332        vcf_reader = self.get_header()
3333        log.debug("Initial header: " + str(vcf_reader.infos))
3334
3335        # Existing annotations
3336        for vcf_annotation in self.get_header().infos:
3337
3338            vcf_annotation_line = self.get_header().infos.get(vcf_annotation)
3339            log.debug(
3340                f"Existing annotations in VCF: {vcf_annotation} [{vcf_annotation_line}]"
3341            )
3342
3343        if annotations:
3344
3345            with TemporaryDirectory(dir=self.get_tmp_dir()) as tmp_dir:
3346
3347                # Export VCF file
3348                tmp_vcf_name = os.path.join(tmp_dir, "input.vcf.gz")
3349
3350                # annotation_bigwig_config
3351                annotation_bigwig_config_list = []
3352
3353                for annotation in annotations:
3354                    annotation_fields = annotations[annotation]
3355
3356                    # Annotation Name
3357                    annotation_name = os.path.basename(annotation)
3358
3359                    if not annotation_fields:
3360                        annotation_fields = {"INFO": None}
3361
3362                    log.debug(f"Annotation '{annotation_name}'")
3363                    log.debug(
3364                        f"Annotation '{annotation_name}' - fields: {annotation_fields}"
3365                    )
3366
3367                    # Create Database
3368                    database = Database(
3369                        database=annotation,
3370                        databases_folders=databases_folders,
3371                        assembly=assembly,
3372                    )
3373
3374                    # Find files
3375                    db_file = database.get_database()
3376                    db_file = full_path(db_file)
3377                    db_hdr_file = database.get_header_file()
3378                    db_hdr_file = full_path(db_hdr_file)
3379                    db_file_type = database.get_format()
3380
3381                    # If db_file is http ?
3382                    if database.get_database().startswith("http"):
3383
3384                        # Datbase is HTTP URL
3385                        db_file_is_http = True
3386
3387                        # DB file keep as URL
3388                        db_file = database.get_database()
3389                        log.warning(
3390                            f"Annotations 'bigwig' database '{db_file}' - is an HTTP URL (experimental)"
3391                        )
3392
3393                        # Retrieve automatic annotation field name
3394                        annotation_field = clean_annotation_field(
3395                            os.path.basename(db_file).replace(".bw", "")
3396                        )
3397                        log.debug(
3398                            f"Create header file with annotation field '{annotation_field}' is an HTTP URL"
3399                        )
3400
3401                        # Create automatic header file
3402                        db_hdr_file = os.path.join(tmp_dir, "header.hdr")
3403                        with open(db_hdr_file, "w") as f:
3404                            f.write("##fileformat=VCFv4.2\n")
3405                            f.write(
3406                                f"""##INFO=<ID={annotation_field},Number=.,Type=Float,Description="{annotation_field} annotation from {db_file}">\n"""
3407                            )
3408                            f.write(f"#CHROM	START	END	{annotation_field}\n")
3409
3410                    else:
3411
3412                        # Datbase is NOT HTTP URL
3413                        db_file_is_http = False
3414
3415                    # Check index - try to create if not exists
3416                    if (
3417                        db_file is None
3418                        or db_hdr_file is None
3419                        or (not os.path.exists(db_file) and not db_file_is_http)
3420                        or not os.path.exists(db_hdr_file)
3421                        or not db_file_type in ["bw"]
3422                    ):
3423                        # if False:
3424                        log.error("Annotation failed: database not valid")
3425                        log.error(f"Annotation annotation file: {db_file}")
3426                        log.error(f"Annotation annotation file type: {db_file_type}")
3427                        log.error(f"Annotation annotation header: {db_hdr_file}")
3428                        raise ValueError(
3429                            f"Annotation failed: database not valid - annotation file {db_file} / annotation file type {db_file_type} / annotation header {db_hdr_file}"
3430                        )
3431                    else:
3432
3433                        # Log
3434                        log.debug(
3435                            f"Annotation '{annotation}' - file: "
3436                            + str(db_file)
3437                            + " and "
3438                            + str(db_hdr_file)
3439                        )
3440
3441                        # Load header as VCF object
3442                        db_hdr_vcf = Variants(input=db_hdr_file)
3443                        db_hdr_vcf_header_infos = db_hdr_vcf.get_header().infos
3444                        log.debug(
3445                            "Annotation database header: "
3446                            + str(db_hdr_vcf_header_infos)
3447                        )
3448
3449                        # For all fields in database
3450                        annotation_fields_full = False
3451                        if "ALL" in annotation_fields or "INFO" in annotation_fields:
3452                            annotation_fields = {
3453                                key: key for key in db_hdr_vcf_header_infos
3454                            }
3455                            log.debug(
3456                                "Annotation database header - All annotations added: "
3457                                + str(annotation_fields)
3458                            )
3459                            annotation_fields_full = True
3460
3461                        # Init
3462                        cyvcf2_header_rename_dict = {}
3463                        cyvcf2_header_list = []
3464                        cyvcf2_header_indexes = {}
3465
3466                        # process annotation fields
3467                        for annotation_field in annotation_fields:
3468
3469                            # New annotation name
3470                            annotation_field_new = annotation_fields[annotation_field]
3471
3472                            # Check annotation field and index in header
3473                            if (
3474                                annotation_field
3475                                in db_hdr_vcf.get_header_columns_as_list()
3476                            ):
3477                                annotation_field_index = (
3478                                    db_hdr_vcf.get_header_columns_as_list().index(
3479                                        annotation_field
3480                                    )
3481                                    - 3
3482                                )
3483                                cyvcf2_header_indexes[annotation_field_new] = (
3484                                    annotation_field_index
3485                                )
3486                            else:
3487                                msg_err = f"Database '{db_file}' does NOT contain annotation field '{annotation_field}'"
3488                                log.error(msg_err)
3489                                raise ValueError(msg_err)
3490
3491                            # Append annotation field in cyvcf2 header list
3492                            cyvcf2_header_rename_dict[annotation_field_new] = (
3493                                db_hdr_vcf_header_infos[annotation_field].id
3494                            )
3495                            cyvcf2_header_list.append(
3496                                {
3497                                    "ID": annotation_field_new,
3498                                    "Number": db_hdr_vcf_header_infos[
3499                                        annotation_field
3500                                    ].num,
3501                                    "Type": db_hdr_vcf_header_infos[
3502                                        annotation_field
3503                                    ].type,
3504                                    "Description": db_hdr_vcf_header_infos[
3505                                        annotation_field
3506                                    ].desc,
3507                                }
3508                            )
3509
3510                            # Add header on VCF
3511                            vcf_reader.infos[annotation_field_new] = vcf.parser._Info(
3512                                annotation_field_new,
3513                                db_hdr_vcf_header_infos[annotation_field].num,
3514                                db_hdr_vcf_header_infos[annotation_field].type,
3515                                db_hdr_vcf_header_infos[annotation_field].desc,
3516                                "HOWARD BigWig annotation",
3517                                "unknown",
3518                                self.code_type_map[
3519                                    db_hdr_vcf_header_infos[annotation_field].type
3520                                ],
3521                            )
3522
3523                        # Load bigwig database
3524                        bw_db = pyBigWig.open(db_file)
3525                        if bw_db.isBigWig():
3526                            log.debug(f"Database '{db_file}' is in 'BigWig' format")
3527                        else:
3528                            msg_err = f"Database '{db_file}' is NOT in 'BigWig' format"
3529                            log.error(msg_err)
3530                            raise ValueError(msg_err)
3531
3532                        annotation_bigwig_config_list.append(
3533                            {
3534                                "db_file": db_file,
3535                                "bw_db": bw_db,
3536                                "cyvcf2_header_rename_dict": cyvcf2_header_rename_dict,
3537                                "cyvcf2_header_list": cyvcf2_header_list,
3538                                "cyvcf2_header_indexes": cyvcf2_header_indexes,
3539                            }
3540                        )
3541
3542                # Annotate
3543                if annotation_bigwig_config_list:
3544
3545                    # Annotation config
3546                    log.debug(
3547                        f"annotation_bigwig_config={annotation_bigwig_config_list}"
3548                    )
3549
3550                    # Export VCF file
3551                    self.export_variant_vcf(
3552                        vcf_file=tmp_vcf_name,
3553                        remove_info=True,
3554                        add_samples=False,
3555                        index=True,
3556                    )
3557
3558                    # Load input tmp file
3559                    input_vcf = cyvcf2.VCF(tmp_vcf_name)
3560
3561                    # Add header in input file
3562                    for annotation_bigwig_config in annotation_bigwig_config_list:
3563                        for cyvcf2_header_field in annotation_bigwig_config.get(
3564                            "cyvcf2_header_list", []
3565                        ):
3566                            log.info(
3567                                f"Annotations 'bigwig' database '{os.path.basename(annotation_bigwig_config.get('db_file'))}' - annotation field '{annotation_bigwig_config.get('cyvcf2_header_rename_dict',{}).get(cyvcf2_header_field.get('ID','Unknown'))}' -> '{cyvcf2_header_field.get('ID')}'"
3568                            )
3569                            input_vcf.add_info_to_header(cyvcf2_header_field)
3570
3571                    # Create output VCF file
3572                    output_vcf_file = os.path.join(tmp_dir, "output.vcf.gz")
3573                    output_vcf = cyvcf2.Writer(output_vcf_file, input_vcf)
3574
3575                    # Fetch variants
3576                    log.info(f"Annotations 'bigwig' start...")
3577                    for variant in input_vcf:
3578
3579                        for annotation_bigwig_config in annotation_bigwig_config_list:
3580
3581                            # DB and indexes
3582                            bw_db = annotation_bigwig_config.get("bw_db", None)
3583                            cyvcf2_header_indexes = annotation_bigwig_config.get(
3584                                "cyvcf2_header_indexes", None
3585                            )
3586
3587                            # Retrieve value from chrom pos
3588                            res = bw_db.values(
3589                                variant.CHROM, variant.POS - 1, variant.POS
3590                            )
3591
3592                            # For each annotation fields (and indexes)
3593                            for cyvcf2_header_index in cyvcf2_header_indexes:
3594
3595                                # If value is NOT nNone
3596                                if not np.isnan(
3597                                    res[cyvcf2_header_indexes[cyvcf2_header_index]]
3598                                ):
3599                                    variant.INFO[cyvcf2_header_index] = res[
3600                                        cyvcf2_header_indexes[cyvcf2_header_index]
3601                                    ]
3602
3603                        # Add record in output file
3604                        output_vcf.write_record(variant)
3605
3606                    # Log
3607                    log.debug(f"Annotation done.")
3608
3609                    # Close and write file
3610                    log.info(f"Annotations 'bigwig' write...")
3611                    output_vcf.close()
3612                    log.debug(f"Write done.")
3613
3614                    # Update variants
3615                    log.info(f"Annotations 'bigwig' update...")
3616                    self.update_from_vcf(output_vcf_file)
3617                    log.debug(f"Update done.")
3618
3619        return True

The function annotation_bigwig annotates variants in a VCF file using bigwig databases.

Parameters

threads: The threads parameter in the annotation_bigwig method is used to specify the number of threads to be used for parallel processing during the annotation process. If the threads parameter is not provided, the method will attempt to determine the optimal number of threads to use based on the system configuration

Returns

True

def annotation_snpsift(self, threads: int = None) -> None: View Source

3621    def annotation_snpsift(self, threads: int = None) -> None:
3622        """
3623        This function annotate with bcftools
3624
3625        :param threads: Number of threads to use
3626        :return: the value of the variable "return_value".
3627        """
3628
3629        # DEBUG
3630        log.debug("Start annotation with bcftools databases")
3631
3632        # Threads
3633        if not threads:
3634            threads = self.get_threads()
3635        log.debug("Threads: " + str(threads))
3636
3637        # Config
3638        config = self.get_config()
3639        log.debug("Config: " + str(config))
3640
3641        # Config - snpSift
3642        snpsift_bin_command = get_bin_command(
3643            bin="SnpSift.jar",
3644            tool="snpsift",
3645            bin_type="jar",
3646            config=config,
3647            default_folder=f"{DEFAULT_TOOLS_FOLDER}/snpeff",
3648        )
3649        if not snpsift_bin_command:
3650            msg_err = f"Annotation failed: no snpsift bin '{snpsift_bin_command}'"
3651            log.error(msg_err)
3652            raise ValueError(msg_err)
3653
3654        # Config - bcftools
3655        bcftools_bin_command = get_bin_command(
3656            bin="bcftools",
3657            tool="bcftools",
3658            bin_type="bin",
3659            config=config,
3660            default_folder=f"{DEFAULT_TOOLS_FOLDER}/bcftools",
3661        )
3662        if not bcftools_bin_command:
3663            msg_err = f"Annotation failed: no bcftools bin '{bcftools_bin_command}'"
3664            log.error(msg_err)
3665            raise ValueError(msg_err)
3666
3667        # Config - BCFTools databases folders
3668        databases_folders = set(
3669            self.get_config()
3670            .get("folders", {})
3671            .get("databases", {})
3672            .get("annotations", ["."])
3673            + self.get_config()
3674            .get("folders", {})
3675            .get("databases", {})
3676            .get("bcftools", ["."])
3677        )
3678        log.debug("Databases annotations: " + str(databases_folders))
3679
3680        # Param
3681        annotations = (
3682            self.get_param()
3683            .get("annotation", {})
3684            .get("snpsift", {})
3685            .get("annotations", None)
3686        )
3687        log.debug("Annotations: " + str(annotations))
3688
3689        # Assembly
3690        assembly = self.get_param().get(
3691            "assembly", self.get_config().get("assembly", DEFAULT_ASSEMBLY)
3692        )
3693
3694        # Data
3695        table_variants = self.get_table_variants()
3696
3697        # Check if not empty
3698        log.debug("Check if not empty")
3699        sql_query_chromosomes = (
3700            f"""SELECT count(*) as count FROM {table_variants} as table_variants"""
3701        )
3702        sql_query_chromosomes_df = self.get_query_to_df(sql_query_chromosomes)
3703        if not sql_query_chromosomes_df["count"][0]:
3704            log.info(f"VCF empty")
3705            return
3706
3707        # VCF header
3708        vcf_reader = self.get_header()
3709        log.debug("Initial header: " + str(vcf_reader.infos))
3710
3711        # Existing annotations
3712        for vcf_annotation in self.get_header().infos:
3713
3714            vcf_annotation_line = self.get_header().infos.get(vcf_annotation)
3715            log.debug(
3716                f"Existing annotations in VCF: {vcf_annotation} [{vcf_annotation_line}]"
3717            )
3718
3719        if annotations:
3720
3721            with TemporaryDirectory(dir=self.get_tmp_dir()) as tmp_dir:
3722
3723                # Export VCF file
3724                tmp_vcf_name = os.path.join(tmp_dir, "input.vcf.gz")
3725
3726                # Init
3727                commands = {}
3728
3729                for annotation in annotations:
3730                    annotation_fields = annotations[annotation]
3731
3732                    # Annotation Name
3733                    annotation_name = os.path.basename(annotation)
3734
3735                    if not annotation_fields:
3736                        annotation_fields = {"INFO": None}
3737
3738                    log.debug(f"Annotation '{annotation_name}'")
3739                    log.debug(
3740                        f"Annotation '{annotation_name}' - fields: {annotation_fields}"
3741                    )
3742
3743                    # Create Database
3744                    database = Database(
3745                        database=annotation,
3746                        databases_folders=databases_folders,
3747                        assembly=assembly,
3748                    )
3749
3750                    # Find files
3751                    db_file = database.get_database()
3752                    db_file = full_path(db_file)
3753                    db_hdr_file = database.get_header_file()
3754                    db_hdr_file = full_path(db_hdr_file)
3755                    db_file_type = database.get_format()
3756                    db_tbi_file = f"{db_file}.tbi"
3757                    db_file_compressed = database.is_compressed()
3758
3759                    # Check if compressed
3760                    if not db_file_compressed:
3761                        log.error(
3762                            f"Annotation '{annotation}' - {db_file} NOT compressed file"
3763                        )
3764                        raise ValueError(
3765                            f"Annotation '{annotation}' - {db_file} NOT compressed file"
3766                        )
3767
3768                    # Check if indexed
3769                    if not os.path.exists(db_tbi_file):
3770                        log.error(
3771                            f"Annotation '{annotation}' - {db_file} NOT indexed file"
3772                        )
3773                        raise ValueError(
3774                            f"Annotation '{annotation}' - {db_file} NOT indexed file"
3775                        )
3776
3777                    # Check index - try to create if not exists
3778                    if not os.path.exists(db_file) or not os.path.exists(db_hdr_file):
3779                        log.error("Annotation failed: database not valid")
3780                        log.error(f"Annotation annotation file: {db_file}")
3781                        log.error(f"Annotation annotation header: {db_hdr_file}")
3782                        log.error(f"Annotation annotation index: {db_tbi_file}")
3783                        raise ValueError(
3784                            f"Annotation failed: database not valid - annotation file {db_file} / annotation header {db_hdr_file} / annotation index {db_tbi_file} / annotation compression {db_file_compressed}"
3785                        )
3786                    else:
3787
3788                        log.debug(
3789                            f"Annotation '{annotation}' - file: "
3790                            + str(db_file)
3791                            + " and "
3792                            + str(db_hdr_file)
3793                        )
3794
3795                        # Load header as VCF object
3796                        db_hdr_vcf = Variants(input=db_hdr_file)
3797                        db_hdr_vcf_header_infos = db_hdr_vcf.get_header().infos
3798                        log.debug(
3799                            "Annotation database header: "
3800                            + str(db_hdr_vcf_header_infos)
3801                        )
3802
3803                        # For all fields in database
3804                        annotation_fields_full = False
3805                        if "ALL" in annotation_fields or "INFO" in annotation_fields:
3806                            annotation_fields = {
3807                                key: key for key in db_hdr_vcf_header_infos
3808                            }
3809                            log.debug(
3810                                "Annotation database header - All annotations added: "
3811                                + str(annotation_fields)
3812                            )
3813                            annotation_fields_full = True
3814
3815                        # # Create file for field rename
3816                        # log.debug("Create file for field rename")
3817                        # tmp_rename = NamedTemporaryFile(
3818                        #     prefix=self.get_prefix(),
3819                        #     dir=self.get_tmp_dir(),
3820                        #     suffix=".rename",
3821                        #     delete=False,
3822                        # )
3823                        # tmp_rename_name = tmp_rename.name
3824                        # tmp_files.append(tmp_rename_name)
3825
3826                        # Number of fields
3827                        nb_annotation_field = 0
3828                        annotation_list = []
3829                        annotation_infos_rename_list = []
3830
3831                        for annotation_field in annotation_fields:
3832
3833                            # field new name, if parametered SKIPPED !!!!!! not managed actually TODO
3834                            annotation_fields_new_name = annotation_fields.get(
3835                                annotation_field, annotation_field
3836                            )
3837                            if not annotation_fields_new_name:
3838                                annotation_fields_new_name = annotation_field
3839
3840                            # Check if field is in DB and if field is not elready in input data
3841                            if (
3842                                annotation_field in db_hdr_vcf.get_header().infos
3843                                and annotation_fields_new_name
3844                                not in self.get_header().infos
3845                            ):
3846
3847                                log.info(
3848                                    f"Annotation '{annotation_name}' - '{annotation_field}' -> '{annotation_fields_new_name}'"
3849                                )
3850
3851                                # BCFTools annotate param to rename fields
3852                                if annotation_field != annotation_fields_new_name:
3853                                    annotation_infos_rename_list.append(
3854                                        f"{annotation_fields_new_name}:=INFO/{annotation_field}"
3855                                    )
3856
3857                                # Add INFO field to header
3858                                db_hdr_vcf_header_infos_number = (
3859                                    db_hdr_vcf_header_infos[annotation_field].num or "."
3860                                )
3861                                db_hdr_vcf_header_infos_type = (
3862                                    db_hdr_vcf_header_infos[annotation_field].type
3863                                    or "String"
3864                                )
3865                                db_hdr_vcf_header_infos_description = (
3866                                    db_hdr_vcf_header_infos[annotation_field].desc
3867                                    or f"{annotation_field} description"
3868                                )
3869                                db_hdr_vcf_header_infos_source = (
3870                                    db_hdr_vcf_header_infos[annotation_field].source
3871                                    or "unknown"
3872                                )
3873                                db_hdr_vcf_header_infos_version = (
3874                                    db_hdr_vcf_header_infos[annotation_field].version
3875                                    or "unknown"
3876                                )
3877
3878                                vcf_reader.infos[annotation_fields_new_name] = (
3879                                    vcf.parser._Info(
3880                                        annotation_fields_new_name,
3881                                        db_hdr_vcf_header_infos_number,
3882                                        db_hdr_vcf_header_infos_type,
3883                                        db_hdr_vcf_header_infos_description,
3884                                        db_hdr_vcf_header_infos_source,
3885                                        db_hdr_vcf_header_infos_version,
3886                                        self.code_type_map[
3887                                            db_hdr_vcf_header_infos_type
3888                                        ],
3889                                    )
3890                                )
3891
3892                                annotation_list.append(annotation_field)
3893
3894                                nb_annotation_field += 1
3895
3896                            else:
3897
3898                                if (
3899                                    annotation_field
3900                                    not in db_hdr_vcf.get_header().infos
3901                                ):
3902                                    log.warning(
3903                                        f"Annotation '{annotation_name}' - '{annotation_field}' - not available in vcf/bed file"
3904                                    )
3905                                if (
3906                                    annotation_fields_new_name
3907                                    in self.get_header().infos
3908                                ):
3909                                    log.warning(
3910                                        f"Annotation '{annotation_name}' - '{annotation_fields_new_name}' - already exists (skipped)"
3911                                    )
3912
3913                        log.info(
3914                            f"Annotation '{annotation_name}' - {nb_annotation_field} annotations available in vcf/bed file"
3915                        )
3916
3917                        annotation_infos = ",".join(annotation_list)
3918
3919                        if annotation_infos != "":
3920
3921                            # Annotated VCF (and error file)
3922                            tmp_annotation_vcf_name = os.path.join(
3923                                tmp_dir, os.path.basename(annotation) + ".vcf.gz"
3924                            )
3925                            tmp_annotation_vcf_name_err = (
3926                                tmp_annotation_vcf_name + ".err"
3927                            )
3928
3929                            # Add fields to annotate
3930                            if not annotation_fields_full:
3931                                annotation_infos_option = f"-info {annotation_infos}"
3932                            else:
3933                                annotation_infos_option = ""
3934
3935                            # Info fields rename
3936                            if annotation_infos_rename_list:
3937                                annotation_infos_rename = " -c " + ",".join(
3938                                    annotation_infos_rename_list
3939                                )
3940                            else:
3941                                annotation_infos_rename = ""
3942
3943                            # Annotate command
3944                            command_annotate = f"{snpsift_bin_command} annotate {annotation_infos_option} {db_file} {tmp_vcf_name} | {bcftools_bin_command} annotate --threads={threads} {annotation_infos_rename} -Oz1 -o {tmp_annotation_vcf_name} 2>>{tmp_annotation_vcf_name_err} "
3945
3946                            # Add command
3947                            commands[command_annotate] = tmp_annotation_vcf_name
3948
3949                if commands:
3950
3951                    # Export VCF file
3952                    self.export_variant_vcf(
3953                        vcf_file=tmp_vcf_name,
3954                        remove_info=True,
3955                        add_samples=False,
3956                        index=True,
3957                    )
3958                    shutil.copyfile(tmp_vcf_name, "/tmp/input.vcf")
3959
3960                    # Num command
3961                    nb_command = 0
3962
3963                    # Annotate
3964                    for command_annotate in commands:
3965                        nb_command += 1
3966                        log.info(
3967                            f"Annotation - Annotate [{nb_command}/{len(commands)}]..."
3968                        )
3969                        log.debug(f"command_annotate={command_annotate}")
3970                        run_parallel_commands([command_annotate], threads)
3971
3972                        # Debug
3973                        shutil.copyfile(commands[command_annotate], "/tmp/snpsift.vcf")
3974
3975                        # Update variants
3976                        log.info(
3977                            f"Annotation - Updating [{nb_command}/{len(commands)}]..."
3978                        )
3979                        self.update_from_vcf(commands[command_annotate])

This function annotate with bcftools

Parameters

threads: Number of threads to use

Returns

the value of the variable "return_value".

def annotation_bcftools(self, threads: int = None) -> None: View Source

3981    def annotation_bcftools(self, threads: int = None) -> None:
3982        """
3983        This function annotate with bcftools
3984
3985        :param threads: Number of threads to use
3986        :return: the value of the variable "return_value".
3987        """
3988
3989        # DEBUG
3990        log.debug("Start annotation with bcftools databases")
3991
3992        # Threads
3993        if not threads:
3994            threads = self.get_threads()
3995        log.debug("Threads: " + str(threads))
3996
3997        # Config
3998        config = self.get_config()
3999        log.debug("Config: " + str(config))
4000
4001        # DEBUG
4002        delete_tmp = True
4003        if self.get_config().get("verbosity", "warning") in ["debug"]:
4004            delete_tmp = False
4005            log.debug("Delete tmp files/folders: " + str(delete_tmp))
4006
4007        # Config - BCFTools bin command
4008        bcftools_bin_command = get_bin_command(
4009            bin="bcftools",
4010            tool="bcftools",
4011            bin_type="bin",
4012            config=config,
4013            default_folder=f"{DEFAULT_TOOLS_FOLDER}/bcftools",
4014        )
4015        if not bcftools_bin_command:
4016            msg_err = f"Annotation failed: no bcftools bin '{bcftools_bin_command}'"
4017            log.error(msg_err)
4018            raise ValueError(msg_err)
4019
4020        # Config - BCFTools databases folders
4021        databases_folders = set(
4022            self.get_config()
4023            .get("folders", {})
4024            .get("databases", {})
4025            .get("annotations", ["."])
4026            + self.get_config()
4027            .get("folders", {})
4028            .get("databases", {})
4029            .get("bcftools", ["."])
4030        )
4031        log.debug("Databases annotations: " + str(databases_folders))
4032
4033        # Param
4034        annotations = (
4035            self.get_param()
4036            .get("annotation", {})
4037            .get("bcftools", {})
4038            .get("annotations", None)
4039        )
4040        log.debug("Annotations: " + str(annotations))
4041
4042        # Assembly
4043        assembly = self.get_param().get(
4044            "assembly", self.get_config().get("assembly", DEFAULT_ASSEMBLY)
4045        )
4046
4047        # Data
4048        table_variants = self.get_table_variants()
4049
4050        # Check if not empty
4051        log.debug("Check if not empty")
4052        sql_query_chromosomes = (
4053            f"""SELECT count(*) as count FROM {table_variants} as table_variants"""
4054        )
4055        sql_query_chromosomes_df = self.get_query_to_df(sql_query_chromosomes)
4056        if not sql_query_chromosomes_df["count"][0]:
4057            log.info(f"VCF empty")
4058            return
4059
4060        # Export in VCF
4061        log.debug("Create initial file to annotate")
4062        tmp_vcf = NamedTemporaryFile(
4063            prefix=self.get_prefix(),
4064            dir=self.get_tmp_dir(),
4065            suffix=".vcf.gz",
4066            delete=False,
4067        )
4068        tmp_vcf_name = tmp_vcf.name
4069
4070        # VCF header
4071        vcf_reader = self.get_header()
4072        log.debug("Initial header: " + str(vcf_reader.infos))
4073
4074        # Existing annotations
4075        for vcf_annotation in self.get_header().infos:
4076
4077            vcf_annotation_line = self.get_header().infos.get(vcf_annotation)
4078            log.debug(
4079                f"Existing annotations in VCF: {vcf_annotation} [{vcf_annotation_line}]"
4080            )
4081
4082        if annotations:
4083
4084            tmp_ann_vcf_list = []
4085            commands = []
4086            tmp_files = []
4087            err_files = []
4088
4089            for annotation in annotations:
4090                annotation_fields = annotations[annotation]
4091
4092                # Annotation Name
4093                annotation_name = os.path.basename(annotation)
4094
4095                if not annotation_fields:
4096                    annotation_fields = {"INFO": None}
4097
4098                log.debug(f"Annotation '{annotation_name}'")
4099                log.debug(
4100                    f"Annotation '{annotation_name}' - fields: {annotation_fields}"
4101                )
4102
4103                # Create Database
4104                database = Database(
4105                    database=annotation,
4106                    databases_folders=databases_folders,
4107                    assembly=assembly,
4108                )
4109
4110                # Find files
4111                db_file = database.get_database()
4112                db_file = full_path(db_file)
4113                db_hdr_file = database.get_header_file()
4114                db_hdr_file = full_path(db_hdr_file)
4115                db_file_type = database.get_format()
4116                db_tbi_file = f"{db_file}.tbi"
4117                db_file_compressed = database.is_compressed()
4118
4119                # Check if compressed
4120                if not db_file_compressed:
4121                    log.error(
4122                        f"Annotation '{annotation}' - {db_file} NOT compressed file"
4123                    )
4124                    raise ValueError(
4125                        f"Annotation '{annotation}' - {db_file} NOT compressed file"
4126                    )
4127
4128                # Check if indexed
4129                if not os.path.exists(db_tbi_file):
4130                    log.error(f"Annotation '{annotation}' - {db_file} NOT indexed file")
4131                    raise ValueError(
4132                        f"Annotation '{annotation}' - {db_file} NOT indexed file"
4133                    )
4134
4135                # Check index - try to create if not exists
4136                if not os.path.exists(db_file) or not os.path.exists(db_hdr_file):
4137                    log.error("Annotation failed: database not valid")
4138                    log.error(f"Annotation annotation file: {db_file}")
4139                    log.error(f"Annotation annotation header: {db_hdr_file}")
4140                    log.error(f"Annotation annotation index: {db_tbi_file}")
4141                    raise ValueError(
4142                        f"Annotation failed: database not valid - annotation file {db_file} / annotation header {db_hdr_file} / annotation index {db_tbi_file} / annotation compression {db_file_compressed}"
4143                    )
4144                else:
4145
4146                    log.debug(
4147                        f"Annotation '{annotation}' - file: "
4148                        + str(db_file)
4149                        + " and "
4150                        + str(db_hdr_file)
4151                    )
4152
4153                    # Load header as VCF object
4154                    db_hdr_vcf = Variants(input=db_hdr_file)
4155                    db_hdr_vcf_header_infos = db_hdr_vcf.get_header().infos
4156                    log.debug(
4157                        "Annotation database header: " + str(db_hdr_vcf_header_infos)
4158                    )
4159
4160                    # For all fields in database
4161                    if "ALL" in annotation_fields or "INFO" in annotation_fields:
4162                        annotation_fields = {
4163                            key: key for key in db_hdr_vcf_header_infos
4164                        }
4165                        log.debug(
4166                            "Annotation database header - All annotations added: "
4167                            + str(annotation_fields)
4168                        )
4169
4170                    # Number of fields
4171                    nb_annotation_field = 0
4172                    annotation_list = []
4173
4174                    for annotation_field in annotation_fields:
4175
4176                        # field new name, if parametered SKIPPED !!!!!! not managed actually TODO
4177                        annotation_fields_new_name = annotation_fields.get(
4178                            annotation_field, annotation_field
4179                        )
4180                        if not annotation_fields_new_name:
4181                            annotation_fields_new_name = annotation_field
4182
4183                        # Check if field is in DB and if field is not elready in input data
4184                        if (
4185                            annotation_field in db_hdr_vcf.get_header().infos
4186                            and annotation_fields_new_name
4187                            not in self.get_header().infos
4188                        ):
4189
4190                            log.info(
4191                                f"Annotation '{annotation_name}' - '{annotation_field}' -> '{annotation_fields_new_name}'"
4192                            )
4193
4194                            # Add INFO field to header
4195                            db_hdr_vcf_header_infos_number = (
4196                                db_hdr_vcf_header_infos[annotation_field].num or "."
4197                            )
4198                            db_hdr_vcf_header_infos_type = (
4199                                db_hdr_vcf_header_infos[annotation_field].type
4200                                or "String"
4201                            )
4202                            db_hdr_vcf_header_infos_description = (
4203                                db_hdr_vcf_header_infos[annotation_field].desc
4204                                or f"{annotation_field} description"
4205                            )
4206                            db_hdr_vcf_header_infos_source = (
4207                                db_hdr_vcf_header_infos[annotation_field].source
4208                                or "unknown"
4209                            )
4210                            db_hdr_vcf_header_infos_version = (
4211                                db_hdr_vcf_header_infos[annotation_field].version
4212                                or "unknown"
4213                            )
4214
4215                            vcf_reader.infos[annotation_fields_new_name] = (
4216                                vcf.parser._Info(
4217                                    annotation_fields_new_name,
4218                                    db_hdr_vcf_header_infos_number,
4219                                    db_hdr_vcf_header_infos_type,
4220                                    db_hdr_vcf_header_infos_description,
4221                                    db_hdr_vcf_header_infos_source,
4222                                    db_hdr_vcf_header_infos_version,
4223                                    self.code_type_map[db_hdr_vcf_header_infos_type],
4224                                )
4225                            )
4226
4227                            # annotation_list.append(annotation_field)
4228                            if annotation_field != annotation_fields_new_name:
4229                                annotation_list.append(
4230                                    f"{annotation_fields_new_name}:=INFO/{annotation_field}"
4231                                )
4232                            else:
4233                                annotation_list.append(annotation_field)
4234
4235                            nb_annotation_field += 1
4236
4237                        else:
4238
4239                            if annotation_field not in db_hdr_vcf.get_header().infos:
4240                                log.warning(
4241                                    f"Annotation '{annotation}' - '{annotation_field}' - not available in vcf/bed file"
4242                                )
4243                            if annotation_fields_new_name in self.get_header().infos:
4244                                log.warning(
4245                                    f"Annotation '{annotation}' - '{annotation_fields_new_name}' - already exists (skipped)"
4246                                )
4247
4248                    log.info(
4249                        f"Annotation '{annotation_name}' - {nb_annotation_field} annotations available in vcf/bed file"
4250                    )
4251
4252                    annotation_infos = ",".join(annotation_list)
4253
4254                    if annotation_infos != "":
4255
4256                        # Protect header for bcftools (remove "#CHROM" and variants line)
4257                        log.debug("Protect Header file - remove #CHROM line if exists")
4258                        tmp_header_vcf = NamedTemporaryFile(
4259                            prefix=self.get_prefix(),
4260                            dir=self.get_tmp_dir(),
4261                            suffix=".hdr",
4262                            delete=False,
4263                        )
4264                        tmp_header_vcf_name = tmp_header_vcf.name
4265                        tmp_files.append(tmp_header_vcf_name)
4266                        # Command
4267                        if db_hdr_file.endswith(".gz"):
4268                            command_extract_header = f"zcat < {db_hdr_file} | grep '^##' > {tmp_header_vcf_name}"
4269                        else:
4270                            command_extract_header = f"cat < {db_hdr_file} | grep '^##' > {tmp_header_vcf_name}"
4271                        # Run
4272                        run_parallel_commands([command_extract_header], 1)
4273
4274                        # Find chomosomes
4275                        log.debug("Find chromosomes ")
4276                        sql_query_chromosomes = f"""SELECT table_variants.\"#CHROM\" as CHROM FROM {table_variants} as table_variants GROUP BY table_variants.\"#CHROM\""""
4277                        sql_query_chromosomes_df = self.get_query_to_df(
4278                            sql_query_chromosomes
4279                        )
4280                        chomosomes_list = list(sql_query_chromosomes_df["CHROM"])
4281
4282                        log.debug("Chromosomes found: " + str(list(chomosomes_list)))
4283
4284                        # BED columns in the annotation file
4285                        if db_file_type in ["bed"]:
4286                            annotation_infos = "CHROM,POS,POS," + annotation_infos
4287
4288                        for chrom in chomosomes_list:
4289
4290                            # Create BED on initial VCF
4291                            log.debug("Create BED on initial VCF: " + str(tmp_vcf_name))
4292                            tmp_bed = NamedTemporaryFile(
4293                                prefix=self.get_prefix(),
4294                                dir=self.get_tmp_dir(),
4295                                suffix=".bed",
4296                                delete=False,
4297                            )
4298                            tmp_bed_name = tmp_bed.name
4299                            tmp_files.append(tmp_bed_name)
4300
4301                            # Detecte regions
4302                            log.debug(
4303                                f"Annotation '{annotation}' - Chromosome '{chrom}' - Start detecting regions..."
4304                            )
4305                            window = 1000000
4306                            sql_query_intervals_for_bed = f"""
4307                                SELECT  \"#CHROM\",
4308                                        CASE WHEN \"POS\"-{window}-1 < 0 THEN 0 ELSE \"POS\"-{window}-1 END,
4309                                        \"POS\"+{window}
4310                                FROM {table_variants} as table_variants
4311                                WHERE table_variants.\"#CHROM\" = '{chrom}'
4312                            """
4313                            regions = self.conn.execute(
4314                                sql_query_intervals_for_bed
4315                            ).fetchall()
4316                            merged_regions = merge_regions(regions)
4317                            log.debug(
4318                                f"Annotation '{annotation}' - Chromosome '{chrom}' - Stop detecting regions..."
4319                            )
4320
4321                            header = ["#CHROM", "START", "END"]
4322                            with open(tmp_bed_name, "w") as f:
4323                                # Write the header with tab delimiter
4324                                f.write("\t".join(header) + "\n")
4325                                for d in merged_regions:
4326                                    # Write each data row with tab delimiter
4327                                    f.write("\t".join(map(str, d)) + "\n")
4328
4329                            # Tmp files
4330                            tmp_annotation_vcf = NamedTemporaryFile(
4331                                prefix=self.get_prefix(),
4332                                dir=self.get_tmp_dir(),
4333                                suffix=".vcf.gz",
4334                                delete=False,
4335                            )
4336                            tmp_annotation_vcf_name = tmp_annotation_vcf.name
4337                            tmp_files.append(tmp_annotation_vcf_name)
4338                            tmp_ann_vcf_list.append(f"{tmp_annotation_vcf_name}")
4339                            tmp_annotation_vcf_name_err = (
4340                                tmp_annotation_vcf_name + ".err"
4341                            )
4342                            err_files.append(tmp_annotation_vcf_name_err)
4343
4344                            # Annotate Command
4345                            log.debug(
4346                                f"Annotation '{annotation}' - add bcftools command"
4347                            )
4348
4349                            # Command
4350                            command_annotate = f"{bcftools_bin_command} annotate --pair-logic exact --regions-file={tmp_bed_name} -a {db_file} -h {tmp_header_vcf_name} -c {annotation_infos} {tmp_vcf_name} -o {tmp_annotation_vcf_name} -Oz1 2>>{tmp_annotation_vcf_name_err} && tabix {tmp_annotation_vcf_name} 2>>{tmp_annotation_vcf_name_err} "
4351
4352                            # Add command
4353                            commands.append(command_annotate)
4354
4355            # if some commands
4356            if commands:
4357
4358                # Export VCF file
4359                self.export_variant_vcf(
4360                    vcf_file=tmp_vcf_name,
4361                    remove_info=True,
4362                    add_samples=False,
4363                    index=True,
4364                )
4365
4366                # Threads
4367                # calculate threads for annotated commands
4368                if commands:
4369                    threads_bcftools_annotate = round(threads / len(commands))
4370                else:
4371                    threads_bcftools_annotate = 1
4372
4373                if not threads_bcftools_annotate:
4374                    threads_bcftools_annotate = 1
4375
4376                # Add threads option to bcftools commands
4377                if threads_bcftools_annotate > 1:
4378                    commands_threaded = []
4379                    for command in commands:
4380                        commands_threaded.append(
4381                            command.replace(
4382                                f"{bcftools_bin_command} annotate ",
4383                                f"{bcftools_bin_command} annotate --threads={threads_bcftools_annotate} ",
4384                            )
4385                        )
4386                    commands = commands_threaded
4387
4388                # Command annotation multithreading
4389                log.debug(f"Annotation - Annotation commands: " + str(commands))
4390                log.info(
4391                    f"Annotation - Annotation multithreaded in "
4392                    + str(len(commands))
4393                    + " commands"
4394                )
4395
4396                run_parallel_commands(commands, threads)
4397
4398                # Merge
4399                tmp_ann_vcf_list_cmd = " ".join(tmp_ann_vcf_list)
4400
4401                if tmp_ann_vcf_list_cmd:
4402
4403                    # Tmp file
4404                    tmp_annotate_vcf = NamedTemporaryFile(
4405                        prefix=self.get_prefix(),
4406                        dir=self.get_tmp_dir(),
4407                        suffix=".vcf.gz",
4408                        delete=True,
4409                    )
4410                    tmp_annotate_vcf_name = tmp_annotate_vcf.name
4411                    tmp_annotate_vcf_name_err = tmp_annotate_vcf_name + ".err"
4412                    err_files.append(tmp_annotate_vcf_name_err)
4413
4414                    # Tmp file remove command
4415                    tmp_files_remove_command = ""
4416                    if tmp_files:
4417                        tmp_files_remove_command = " && rm -f " + " ".join(tmp_files)
4418
4419                    # Command merge
4420                    merge_command = f"{bcftools_bin_command} merge --force-samples --threads={threads} {tmp_vcf_name} {tmp_ann_vcf_list_cmd} -o {tmp_annotate_vcf_name} -Oz 2>>{tmp_annotate_vcf_name_err} {tmp_files_remove_command}"
4421                    log.info(
4422                        f"Annotation - Annotation merging "
4423                        + str(len(commands))
4424                        + " annotated files"
4425                    )
4426                    log.debug(f"Annotation - merge command: {merge_command}")
4427                    run_parallel_commands([merge_command], 1)
4428
4429                    # Error messages
4430                    log.info(f"Error/Warning messages:")
4431                    error_message_command_all = []
4432                    error_message_command_warning = []
4433                    error_message_command_err = []
4434                    for err_file in err_files:
4435                        with open(err_file, "r") as f:
4436                            for line in f:
4437                                message = line.strip()
4438                                error_message_command_all.append(message)
4439                                if line.startswith("[W::"):
4440                                    error_message_command_warning.append(message)
4441                                if line.startswith("[E::"):
4442                                    error_message_command_err.append(
4443                                        f"{err_file}: " + message
4444                                    )
4445                    # log info
4446                    for message in list(
4447                        set(error_message_command_err + error_message_command_warning)
4448                    ):
4449                        log.info(f"   {message}")
4450                    # debug info
4451                    for message in list(set(error_message_command_all)):
4452                        log.debug(f"   {message}")
4453                    # failed
4454                    if len(error_message_command_err):
4455                        log.error("Annotation failed: Error in commands")
4456                        raise ValueError("Annotation failed: Error in commands")
4457
4458                    # Update variants
4459                    log.info(f"Annotation - Updating...")
4460                    self.update_from_vcf(tmp_annotate_vcf_name)

This function annotate with bcftools

Parameters

threads: Number of threads to use

Returns

the value of the variable "return_value".

def annotation_exomiser(self, threads: int = None) -> None: View Source

4462    def annotation_exomiser(self, threads: int = None) -> None:
4463        """
4464        This function annotate with Exomiser
4465
4466        This function uses args as parameters, in section "annotation" -> "exomiser", with sections:
4467        - "analysis" (dict/file):
4468            Full analysis dictionnary parameters (see Exomiser docs).
4469            Either a dict, or a file in JSON or YAML format.
4470            These parameters may change depending on other parameters (e.g. phenotipicFeatures/HPO)
4471            Default : None
4472        - "preset" (string):
4473            Analysis preset (available in config folder).
4474            Used if no full "analysis" is provided.
4475            Default: "exome"
4476        - "phenopacket" (dict/file):
4477            Samples and phenotipic features parameters (see Exomiser docs).
4478            Either a dict, or a file in JSON or YAML format.
4479            Default: None
4480        - "subject" (dict):
4481            Sample parameters (see Exomiser docs).
4482            Example:
4483                "subject":
4484                    {
4485                        "id": "ISDBM322017",
4486                        "sex": "FEMALE"
4487                    }
4488            Default: None
4489        - "sample" (string):
4490            Sample name to construct "subject" section:
4491                "subject":
4492                    {
4493                        "id": "<sample>",
4494                        "sex": "UNKNOWN_SEX"
4495                    }
4496            Default: None
4497        - "phenotypicFeatures" (dict)
4498            Phenotypic features to construct "subject" section.
4499            Example:
4500                "phenotypicFeatures":
4501                    [
4502                        { "type": { "id": "HP:0001159", "label": "Syndactyly" } },
4503                        { "type": { "id": "HP:0000486", "label": "Strabismus" } }
4504                    ]
4505        - "hpo" (list)
4506            List of HPO ids as phenotypic features.
4507            Example:
4508                "hpo": ['0001156', '0001363', '0011304', '0010055']
4509            Default: []
4510        - "outputOptions" (dict):
4511            Output options (see Exomiser docs).
4512            Default:
4513                "output_options" =
4514                    {
4515                        "outputContributingVariantsOnly": False,
4516                        "numGenes": 0,
4517                        "outputFormats": ["TSV_VARIANT", "VCF"]
4518                    }
4519        - "transcript_source" (string):
4520            Transcript source (either "refseq", "ucsc", "ensembl")
4521            Default: "refseq"
4522        - "exomiser_to_info" (boolean):
4523            Add exomiser TSV file columns as INFO fields in VCF.
4524            Default: False
4525        - "release" (string):
4526            Exomise database release.
4527            If not exists, database release will be downloaded (take a while).
4528            Default: None (provided by application.properties configuration file)
4529        - "exomiser_application_properties" (file):
4530            Exomiser configuration file (see Exomiser docs).
4531            Useful to automatically download databases (especially for specific genome databases).
4532
4533        Notes:
4534        - If no sample in parameters, first sample in VCF will be chosen
4535        - If no HPO found, "hiPhivePrioritiser" analysis step will be switch off
4536
4537        :param threads: The number of threads to use
4538        :return: None.
4539        """
4540
4541        # DEBUG
4542        log.debug("Start annotation with Exomiser databases")
4543
4544        # Threads
4545        if not threads:
4546            threads = self.get_threads()
4547        log.debug("Threads: " + str(threads))
4548
4549        # Config
4550        config = self.get_config()
4551        log.debug("Config: " + str(config))
4552
4553        # Config - Folders - Databases
4554        databases_folders = (
4555            config.get("folders", {})
4556            .get("databases", {})
4557            .get("exomiser", f"{DEFAULT_DATABASE_FOLDER}/exomiser/current")
4558        )
4559        databases_folders = full_path(databases_folders)
4560        if not os.path.exists(databases_folders):
4561            log.error(f"Databases annotations: {databases_folders} NOT found")
4562        log.debug("Databases annotations: " + str(databases_folders))
4563
4564        # Config - Exomiser
4565        exomiser_bin_command = get_bin_command(
4566            bin="exomiser-cli*.jar",
4567            tool="exomiser",
4568            bin_type="jar",
4569            config=config,
4570            default_folder=f"{DEFAULT_TOOLS_FOLDER}/exomiser",
4571        )
4572        log.debug("Exomiser bin command: " + str(exomiser_bin_command))
4573        if not exomiser_bin_command:
4574            msg_err = f"Annotation failed: no exomiser bin '{exomiser_bin_command}'"
4575            log.error(msg_err)
4576            raise ValueError(msg_err)
4577
4578        # Param
4579        param = self.get_param()
4580        log.debug("Param: " + str(param))
4581
4582        # Param - Exomiser
4583        param_exomiser = param.get("annotation", {}).get("exomiser", {})
4584        log.debug(f"Param Exomiser: {param_exomiser}")
4585
4586        # Param - Assembly
4587        assembly = param.get("assembly", config.get("assembly", DEFAULT_ASSEMBLY))
4588        log.debug("Assembly: " + str(assembly))
4589
4590        # Data
4591        table_variants = self.get_table_variants()
4592
4593        # Check if not empty
4594        log.debug("Check if not empty")
4595        sql_query_chromosomes = (
4596            f"""SELECT count(*) as count FROM {table_variants} as table_variants"""
4597        )
4598        if not self.get_query_to_df(f"{sql_query_chromosomes}")["count"][0]:
4599            log.info(f"VCF empty")
4600            return False
4601
4602        # VCF header
4603        vcf_reader = self.get_header()
4604        log.debug("Initial header: " + str(vcf_reader.infos))
4605
4606        # Samples
4607        samples = self.get_header_sample_list()
4608        if not samples:
4609            log.error("No Samples in VCF")
4610            return False
4611        log.debug(f"Samples: {samples}")
4612
4613        # Memory limit
4614        memory_limit = self.get_memory("8G")
4615        log.debug(f"memory_limit: {memory_limit}")
4616
4617        # Exomiser java options
4618        exomiser_java_options = (
4619            f" -Xmx{memory_limit} -XX:+UseParallelGC -XX:ParallelGCThreads={threads} "
4620        )
4621        log.debug(f"Exomiser java options: {exomiser_java_options}")
4622
4623        # Download Exomiser (if not exists)
4624        exomiser_release = param_exomiser.get("release", None)
4625        exomiser_application_properties = param_exomiser.get(
4626            "exomiser_application_properties", None
4627        )
4628        databases_download_exomiser(
4629            assemblies=[assembly],
4630            exomiser_folder=databases_folders,
4631            exomiser_release=exomiser_release,
4632            exomiser_phenotype_release=exomiser_release,
4633            exomiser_application_properties=exomiser_application_properties,
4634        )
4635
4636        # Force annotation
4637        force_update_annotation = True
4638
4639        if "Exomiser" not in self.get_header().infos or force_update_annotation:
4640            log.debug("Start annotation Exomiser")
4641
4642            with TemporaryDirectory(dir=self.get_tmp_dir()) as tmp_dir:
4643
4644                # tmp_dir = "/tmp/exomiser"
4645
4646                ### ANALYSIS ###
4647                ################
4648
4649                # Create analysis.json through analysis dict
4650                # either analysis in param or by default
4651                # depending on preset exome/genome)
4652
4653                # Init analysis dict
4654                param_exomiser_analysis_dict = {}
4655
4656                # analysis from param
4657                param_exomiser_analysis = param_exomiser.get("analysis", {})
4658                param_exomiser_analysis = full_path(param_exomiser_analysis)
4659
4660                # If analysis in param -> load anlaysis json
4661                if param_exomiser_analysis:
4662
4663                    # If param analysis is a file and exists
4664                    if isinstance(param_exomiser_analysis, str) and os.path.exists(
4665                        param_exomiser_analysis
4666                    ):
4667                        # Load analysis file into analysis dict (either yaml or json)
4668                        with open(param_exomiser_analysis) as json_file:
4669                            param_exomiser_analysis_dict = yaml.safe_load(json_file)
4670
4671                    # If param analysis is a dict
4672                    elif isinstance(param_exomiser_analysis, dict):
4673                        # Load analysis dict into analysis dict (either yaml or json)
4674                        param_exomiser_analysis_dict = param_exomiser_analysis
4675
4676                    # Error analysis type
4677                    else:
4678                        log.error(f"Analysis type unknown. Check param file.")
4679                        raise ValueError(f"Analysis type unknown. Check param file.")
4680
4681                # Case no input analysis config file/dict
4682                # Use preset (exome/genome) to open default config file
4683                if not param_exomiser_analysis_dict:
4684
4685                    # default preset
4686                    default_preset = "exome"
4687
4688                    # Get param preset or default preset
4689                    param_exomiser_preset = param_exomiser.get("preset", default_preset)
4690
4691                    # Try to find if preset is a file
4692                    if os.path.exists(param_exomiser_preset):
4693                        # Preset file is provided in full path
4694                        param_exomiser_analysis_default_config_file = (
4695                            param_exomiser_preset
4696                        )
4697                    # elif os.path.exists(full_path(param_exomiser_preset)):
4698                    #     # Preset file is provided in full path
4699                    #     param_exomiser_analysis_default_config_file = full_path(param_exomiser_preset)
4700                    elif os.path.exists(
4701                        os.path.join(folder_config, param_exomiser_preset)
4702                    ):
4703                        # Preset file is provided a basename in config folder (can be a path with subfolders)
4704                        param_exomiser_analysis_default_config_file = os.path.join(
4705                            folder_config, param_exomiser_preset
4706                        )
4707                    else:
4708                        # Construct preset file
4709                        param_exomiser_analysis_default_config_file = os.path.join(
4710                            folder_config,
4711                            f"preset-{param_exomiser_preset}-analysis.json",
4712                        )
4713
4714                    # If preset file exists
4715                    param_exomiser_analysis_default_config_file = full_path(
4716                        param_exomiser_analysis_default_config_file
4717                    )
4718                    if os.path.exists(param_exomiser_analysis_default_config_file):
4719                        # Load prest file into analysis dict (either yaml or json)
4720                        with open(
4721                            param_exomiser_analysis_default_config_file
4722                        ) as json_file:
4723                            param_exomiser_analysis_dict["analysis"] = yaml.safe_load(
4724                                json_file
4725                            )
4726
4727                    # Error preset file
4728                    else:
4729                        log.error(
4730                            f"No analysis preset config file ({param_exomiser_analysis_default_config_file})"
4731                        )
4732                        raise ValueError(
4733                            f"No analysis preset config file ({param_exomiser_analysis_default_config_file})"
4734                        )
4735
4736                # If no analysis dict created
4737                if not param_exomiser_analysis_dict:
4738                    log.error(f"No analysis config")
4739                    raise ValueError(f"No analysis config")
4740
4741                # Log
4742                log.debug(f"Pre analysis dict: {param_exomiser_analysis_dict}")
4743
4744                ### PHENOPACKET ###
4745                ###################
4746
4747                # If no PhenoPacket in analysis dict -> check in param
4748                if "phenopacket" not in param_exomiser_analysis_dict:
4749
4750                    # If PhenoPacket in param -> load anlaysis json
4751                    if param_exomiser.get("phenopacket", None):
4752
4753                        param_exomiser_phenopacket = param_exomiser.get("phenopacket")
4754                        param_exomiser_phenopacket = full_path(
4755                            param_exomiser_phenopacket
4756                        )
4757
4758                        # If param phenopacket is a file and exists
4759                        if isinstance(
4760                            param_exomiser_phenopacket, str
4761                        ) and os.path.exists(param_exomiser_phenopacket):
4762                            # Load phenopacket file into analysis dict (either yaml or json)
4763                            with open(param_exomiser_phenopacket) as json_file:
4764                                param_exomiser_analysis_dict["phenopacket"] = (
4765                                    yaml.safe_load(json_file)
4766                                )
4767
4768                        # If param phenopacket is a dict
4769                        elif isinstance(param_exomiser_phenopacket, dict):
4770                            # Load phenopacket dict into analysis dict (either yaml or json)
4771                            param_exomiser_analysis_dict["phenopacket"] = (
4772                                param_exomiser_phenopacket
4773                            )
4774
4775                        # Error phenopacket type
4776                        else:
4777                            log.error(f"Phenopacket type unknown. Check param file.")
4778                            raise ValueError(
4779                                f"Phenopacket type unknown. Check param file."
4780                            )
4781
4782                # If no PhenoPacket in analysis dict -> construct from sample and HPO in param
4783                if "phenopacket" not in param_exomiser_analysis_dict:
4784
4785                    # Init PhenoPacket
4786                    param_exomiser_analysis_dict["phenopacket"] = {
4787                        "id": "analysis",
4788                        "proband": {},
4789                    }
4790
4791                    ### Add subject ###
4792
4793                    # If subject exists
4794                    param_exomiser_subject = param_exomiser.get("subject", {})
4795
4796                    # If subject not exists -> found sample ID
4797                    if not param_exomiser_subject:
4798
4799                        # Found sample ID in param
4800                        sample = param_exomiser.get("sample", None)
4801
4802                        # Find sample ID (first sample)
4803                        if not sample:
4804                            sample_list = self.get_header_sample_list()
4805                            if len(sample_list) > 0:
4806                                sample = sample_list[0]
4807                            else:
4808                                log.error(f"No sample found")
4809                                raise ValueError(f"No sample found")
4810
4811                        # Create subject
4812                        param_exomiser_subject = {"id": sample, "sex": "UNKNOWN_SEX"}
4813
4814                    # Add to dict
4815                    param_exomiser_analysis_dict["phenopacket"][
4816                        "subject"
4817                    ] = param_exomiser_subject
4818
4819                    ### Add "phenotypicFeatures" ###
4820
4821                    # If phenotypicFeatures exists
4822                    param_exomiser_phenotypicfeatures = param_exomiser.get(
4823                        "phenotypicFeatures", []
4824                    )
4825
4826                    # If phenotypicFeatures not exists -> Try to infer from hpo list
4827                    if not param_exomiser_phenotypicfeatures:
4828
4829                        # Found HPO in param
4830                        param_exomiser_hpo = param_exomiser.get("hpo", [])
4831
4832                        # Split HPO if list in string format separated by comma
4833                        if isinstance(param_exomiser_hpo, str):
4834                            param_exomiser_hpo = param_exomiser_hpo.split(",")
4835
4836                        # Create HPO list
4837                        for hpo in param_exomiser_hpo:
4838                            hpo_clean = re.sub("[^0-9]", "", hpo)
4839                            param_exomiser_phenotypicfeatures.append(
4840                                {
4841                                    "type": {
4842                                        "id": f"HP:{hpo_clean}",
4843                                        "label": f"HP:{hpo_clean}",
4844                                    }
4845                                }
4846                            )
4847
4848                    # Add to dict
4849                    param_exomiser_analysis_dict["phenopacket"][
4850                        "phenotypicFeatures"
4851                    ] = param_exomiser_phenotypicfeatures
4852
4853                    # If phenotypicFeatures not exists -> Remove hiPhivePrioritiser step
4854                    if not param_exomiser_phenotypicfeatures:
4855                        for step in param_exomiser_analysis_dict.get(
4856                            "analysis", {}
4857                        ).get("steps", []):
4858                            if "hiPhivePrioritiser" in step:
4859                                param_exomiser_analysis_dict.get("analysis", {}).get(
4860                                    "steps", []
4861                                ).remove(step)
4862
4863                ### Add Input File ###
4864
4865                # Initial file name and htsFiles
4866                tmp_vcf_name = os.path.join(tmp_dir, "initial.vcf.gz")
4867                param_exomiser_analysis_dict["phenopacket"]["htsFiles"] = [
4868                    {
4869                        "uri": tmp_vcf_name,
4870                        "htsFormat": "VCF",
4871                        "genomeAssembly": assembly,
4872                    }
4873                ]
4874
4875                ### Add metaData ###
4876
4877                # If metaData not in analysis dict
4878                if "metaData" not in param_exomiser_analysis_dict:
4879                    param_exomiser_analysis_dict["phenopacket"]["metaData"] = {
4880                        "created": f"{datetime.datetime.now()}".replace(" ", "T") + "Z",
4881                        "createdBy": "howard",
4882                        "phenopacketSchemaVersion": 1,
4883                    }
4884
4885                ### OutputOptions ###
4886
4887                # Init output result folder
4888                output_results = os.path.join(tmp_dir, "results")
4889
4890                # If no outputOptions in analysis dict
4891                if "outputOptions" not in param_exomiser_analysis_dict:
4892
4893                    # default output formats
4894                    defaut_output_formats = ["TSV_VARIANT", "VCF"]
4895
4896                    # Get outputOptions in param
4897                    output_options = param_exomiser.get("outputOptions", None)
4898
4899                    # If no output_options in param -> check
4900                    if not output_options:
4901                        output_options = {
4902                            "outputContributingVariantsOnly": False,
4903                            "numGenes": 0,
4904                            "outputFormats": defaut_output_formats,
4905                        }
4906
4907                    # Replace outputDirectory in output options
4908                    output_options["outputDirectory"] = output_results
4909                    output_options["outputFileName"] = "howard"
4910
4911                    # Add outputOptions in analysis dict
4912                    param_exomiser_analysis_dict["outputOptions"] = output_options
4913
4914                else:
4915
4916                    # Replace output_results and output format (if exists in param)
4917                    param_exomiser_analysis_dict["outputOptions"][
4918                        "outputDirectory"
4919                    ] = output_results
4920                    param_exomiser_analysis_dict["outputOptions"]["outputFormats"] = (
4921                        list(
4922                            set(
4923                                param_exomiser_analysis_dict.get(
4924                                    "outputOptions", {}
4925                                ).get("outputFormats", [])
4926                                + ["TSV_VARIANT", "VCF"]
4927                            )
4928                        )
4929                    )
4930
4931                # log
4932                log.debug(f"Pre analysis dict: {param_exomiser_analysis_dict}")
4933
4934                ### ANALYSIS FILE ###
4935                #####################
4936
4937                ### Full JSON analysis config file ###
4938
4939                exomiser_analysis = os.path.join(tmp_dir, "analysis.json")
4940                with open(exomiser_analysis, "w") as fp:
4941                    json.dump(param_exomiser_analysis_dict, fp, indent=4)
4942
4943                ### SPLIT analysis and sample config files
4944
4945                # Splitted analysis dict
4946                param_exomiser_analysis_dict_for_split = (
4947                    param_exomiser_analysis_dict.copy()
4948                )
4949
4950                # Phenopacket JSON file
4951                exomiser_analysis_phenopacket = os.path.join(
4952                    tmp_dir, "analysis_phenopacket.json"
4953                )
4954                with open(exomiser_analysis_phenopacket, "w") as fp:
4955                    json.dump(
4956                        param_exomiser_analysis_dict_for_split.get("phenopacket"),
4957                        fp,
4958                        indent=4,
4959                    )
4960
4961                # Analysis JSON file without Phenopacket parameters
4962                param_exomiser_analysis_dict_for_split.pop("phenopacket")
4963                exomiser_analysis_analysis = os.path.join(
4964                    tmp_dir, "analysis_analysis.json"
4965                )
4966                with open(exomiser_analysis_analysis, "w") as fp:
4967                    json.dump(param_exomiser_analysis_dict_for_split, fp, indent=4)
4968
4969                ### INITAL VCF file ###
4970                #######################
4971
4972                ### Create list of samples to use and include inti initial VCF file ####
4973
4974                # Subject (main sample)
4975                # Get sample ID in analysis dict
4976                sample_subject = (
4977                    param_exomiser_analysis_dict.get("phenopacket", {})
4978                    .get("subject", {})
4979                    .get("id", None)
4980                )
4981                sample_proband = (
4982                    param_exomiser_analysis_dict.get("phenopacket", {})
4983                    .get("proband", {})
4984                    .get("subject", {})
4985                    .get("id", None)
4986                )
4987                sample = []
4988                if sample_subject:
4989                    sample.append(sample_subject)
4990                if sample_proband:
4991                    sample.append(sample_proband)
4992
4993                # Get sample ID within Pedigree
4994                pedigree_persons_list = (
4995                    param_exomiser_analysis_dict.get("phenopacket", {})
4996                    .get("pedigree", {})
4997                    .get("persons", {})
4998                )
4999
5000                # Create list with all sample ID in pedigree (if exists)
5001                pedigree_persons = []
5002                for person in pedigree_persons_list:
5003                    pedigree_persons.append(person.get("individualId"))
5004
5005                # Concat subject sample ID and samples ID in pedigreesamples
5006                samples = list(set(sample + pedigree_persons))
5007
5008                # Check if sample list is not empty
5009                if not samples:
5010                    log.error(f"No samples found")
5011                    raise ValueError(f"No samples found")
5012
5013                # Create VCF with sample (either sample in param or first one by default)
5014                # Export VCF file
5015                self.export_variant_vcf(
5016                    vcf_file=tmp_vcf_name,
5017                    remove_info=True,
5018                    add_samples=True,
5019                    list_samples=samples,
5020                    index=False,
5021                )
5022
5023                ### Execute Exomiser ###
5024                ########################
5025
5026                # Init command
5027                exomiser_command = ""
5028
5029                # Command exomiser options
5030                exomiser_options = f" --spring.config.location={databases_folders}/{assembly}/application.properties --exomiser.data-directory={databases_folders}/{assembly} "
5031
5032                # Release
5033                exomiser_release = param_exomiser.get("release", None)
5034                if exomiser_release:
5035                    # phenotype data version
5036                    exomiser_options += (
5037                        f" --exomiser.phenotype.data-version={exomiser_release} "
5038                    )
5039                    # data version
5040                    exomiser_options += (
5041                        f" --exomiser.{assembly}.data-version={exomiser_release} "
5042                    )
5043                    # variant white list
5044                    variant_white_list_file = (
5045                        f"{exomiser_release}_{assembly}_clinvar_whitelist.tsv.gz"
5046                    )
5047                    if os.path.exists(
5048                        os.path.join(
5049                            databases_folders, assembly, variant_white_list_file
5050                        )
5051                    ):
5052                        exomiser_options += f" --exomiser.{assembly}.variant-white-list-path={variant_white_list_file} "
5053
5054                # transcript_source
5055                transcript_source = param_exomiser.get(
5056                    "transcript_source", None
5057                )  # ucsc, refseq, ensembl
5058                if transcript_source:
5059                    exomiser_options += (
5060                        f" --exomiser.{assembly}.transcript-source={transcript_source} "
5061                    )
5062
5063                # If analysis contain proband param
5064                if param_exomiser_analysis_dict.get("phenopacket", {}).get(
5065                    "proband", {}
5066                ):
5067                    exomiser_command_analysis = f" {exomiser_bin_command} --analysis={exomiser_analysis_analysis} --sample={exomiser_analysis_phenopacket} {exomiser_options} "
5068
5069                # If no proband (usually uniq sample)
5070                else:
5071                    exomiser_command_analysis = f" {exomiser_bin_command} --analysis={exomiser_analysis} {exomiser_options}"
5072
5073                # Log
5074                log.debug(f"exomiser_command_analysis={exomiser_command_analysis}")
5075
5076                # Run command
5077                result = subprocess.call(
5078                    exomiser_command_analysis.split(), stdout=subprocess.PIPE
5079                )
5080                if result:
5081                    log.error("Exomiser command failed")
5082                    raise ValueError("Exomiser command failed")
5083
5084                ### RESULTS ###
5085                ###############
5086
5087                ### Annotate with TSV fields ###
5088
5089                # Init result tsv file
5090                exomiser_to_info = param_exomiser.get("exomiser_to_info", False)
5091
5092                # Init result tsv file
5093                output_results_tsv = os.path.join(output_results, "howard.variants.tsv")
5094
5095                # Parse TSV file and explode columns in INFO field
5096                if exomiser_to_info and os.path.exists(output_results_tsv):
5097
5098                    # Log
5099                    log.debug("Exomiser columns to VCF INFO field")
5100
5101                    # Retrieve columns and types
5102                    query = f""" SELECT * FROM read_csv('{output_results_tsv}', auto_detect=True, delim='\t', sample_size=-1) LIMIT 0 """
5103                    output_results_tsv_df = self.get_query_to_df(query)
5104                    output_results_tsv_columns = output_results_tsv_df.columns.tolist()
5105
5106                    # Init concat fields for update
5107                    sql_query_update_concat_fields = []
5108
5109                    # Fields to avoid
5110                    fields_to_avoid = [
5111                        "CONTIG",
5112                        "START",
5113                        "END",
5114                        "REF",
5115                        "ALT",
5116                        "QUAL",
5117                        "FILTER",
5118                        "GENOTYPE",
5119                    ]
5120
5121                    # List all columns to add into header
5122                    for header_column in output_results_tsv_columns:
5123
5124                        # If header column is enable
5125                        if header_column not in fields_to_avoid:
5126
5127                            # Header info type
5128                            header_info_type = "String"
5129                            header_column_df = output_results_tsv_df[header_column]
5130                            header_column_df_dtype = header_column_df.dtype
5131                            if header_column_df_dtype == object:
5132                                if (
5133                                    pd.to_numeric(header_column_df, errors="coerce")
5134                                    .notnull()
5135                                    .all()
5136                                ):
5137                                    header_info_type = "Float"
5138                            else:
5139                                header_info_type = "Integer"
5140
5141                            # Header info
5142                            characters_to_validate = ["-"]
5143                            pattern = "[" + "".join(characters_to_validate) + "]"
5144                            header_info_name = re.sub(
5145                                pattern,
5146                                "_",
5147                                f"Exomiser_{header_column}".replace("#", ""),
5148                            )
5149                            header_info_number = "."
5150                            header_info_description = (
5151                                f"Exomiser {header_column} annotation"
5152                            )
5153                            header_info_source = "Exomiser"
5154                            header_info_version = "unknown"
5155                            header_info_code = CODE_TYPE_MAP[header_info_type]
5156                            vcf_reader.infos[header_info_name] = vcf.parser._Info(
5157                                header_info_name,
5158                                header_info_number,
5159                                header_info_type,
5160                                header_info_description,
5161                                header_info_source,
5162                                header_info_version,
5163                                header_info_code,
5164                            )
5165
5166                            # Add field to add for update to concat fields
5167                            sql_query_update_concat_fields.append(
5168                                f"""
5169                                CASE
5170                                    WHEN table_parquet."{header_column}" NOT IN ('','.')
5171                                    THEN concat(
5172                                        '{header_info_name}=',
5173                                        table_parquet."{header_column}",
5174                                        ';'
5175                                        )
5176
5177                                    ELSE ''
5178                                END
5179                            """
5180                            )
5181
5182                    # Update query
5183                    sql_query_update = f"""
5184                        UPDATE {table_variants} as table_variants
5185                            SET INFO = concat(
5186                                            CASE
5187                                                WHEN INFO NOT IN ('', '.')
5188                                                THEN INFO
5189                                                ELSE ''
5190                                            END,
5191                                            CASE
5192                                                WHEN table_variants.INFO NOT IN ('','.')
5193                                                THEN ';'
5194                                                ELSE ''
5195                                            END,
5196                                            (
5197                                            SELECT 
5198                                                concat(
5199                                                    {",".join(sql_query_update_concat_fields)}
5200                                                )
5201                                            FROM read_csv('{output_results_tsv}', auto_detect=True, delim='\t', sample_size=-1) as table_parquet
5202                                                    WHERE concat('chr', CAST(table_parquet.\"CONTIG\" AS STRING)) = table_variants.\"#CHROM\"
5203                                                    AND table_parquet.\"START\" = table_variants.\"POS\"
5204                                                    AND table_parquet.\"ALT\" = table_variants.\"ALT\"
5205                                                    AND table_parquet.\"REF\" = table_variants.\"REF\"
5206                                            )
5207                                        )
5208                            ;
5209                        """
5210
5211                    # Update
5212                    self.conn.execute(sql_query_update)
5213
5214                ### Annotate with VCF INFO field ###
5215
5216                # Init result VCF file
5217                output_results_vcf = os.path.join(output_results, "howard.vcf.gz")
5218
5219                # If VCF exists
5220                if os.path.exists(output_results_vcf):
5221
5222                    # Log
5223                    log.debug("Exomiser result VCF update variants")
5224
5225                    # Find Exomiser INFO field annotation in header
5226                    with gzip.open(output_results_vcf, "rt") as f:
5227                        header_list = self.read_vcf_header(f)
5228                    exomiser_vcf_header = vcf.Reader(
5229                        io.StringIO("\n".join(header_list))
5230                    )
5231
5232                    # Add annotation INFO field to header
5233                    vcf_reader.infos["Exomiser"] = exomiser_vcf_header.infos["Exomiser"]
5234
5235                    # Update variants with VCF
5236                    self.update_from_vcf(output_results_vcf)
5237
5238        return True

This function annotate with Exomiser

This function uses args as parameters, in section "annotation" -> "exomiser", with sections:

"analysis" (dict/file): Full analysis dictionnary parameters (see Exomiser docs). Either a dict, or a file in JSON or YAML format. These parameters may change depending on other parameters (e.g. phenotipicFeatures/HPO) Default : None
"preset" (string): Analysis preset (available in config folder). Used if no full "analysis" is provided. Default: "exome"
"phenopacket" (dict/file): Samples and phenotipic features parameters (see Exomiser docs). Either a dict, or a file in JSON or YAML format. Default: None
"subject" (dict): Sample parameters (see Exomiser docs). Example: "subject": { "id": "ISDBM322017", "sex": "FEMALE" } Default: None
"sample" (string): Sample name to construct "subject" section: "subject": { "id": "", "sex": "UNKNOWN_SEX" } Default: None
"phenotypicFeatures" (dict) Phenotypic features to construct "subject" section. Example: "phenotypicFeatures": [ { "type": { "id": "HP:0001159", "label": "Syndactyly" } }, { "type": { "id": "HP:0000486", "label": "Strabismus" } } ]
"hpo" (list) List of HPO ids as phenotypic features. Example: "hpo": ['0001156', '0001363', '0011304', '0010055'] Default: []
"outputOptions" (dict): Output options (see Exomiser docs). Default: "output_options" = { "outputContributingVariantsOnly": False, "numGenes": 0, "outputFormats": ["TSV_VARIANT", "VCF"] }
"transcript_source" (string): Transcript source (either "refseq", "ucsc", "ensembl") Default: "refseq"
"exomiser_to_info" (boolean): Add exomiser TSV file columns as INFO fields in VCF. Default: False
"release" (string): Exomise database release. If not exists, database release will be downloaded (take a while). Default: None (provided by application.properties configuration file)
"exomiser_application_properties" (file): Exomiser configuration file (see Exomiser docs). Useful to automatically download databases (especially for specific genome databases).

Notes:

If no sample in parameters, first sample in VCF will be chosen
If no HPO found, "hiPhivePrioritiser" analysis step will be switch off

Parameters

threads: The number of threads to use

Returns

None.

def annotation_snpeff(self, threads: int = None) -> None: View Source

5240    def annotation_snpeff(self, threads: int = None) -> None:
5241        """
5242        This function annotate with snpEff
5243
5244        :param threads: The number of threads to use
5245        :return: the value of the variable "return_value".
5246        """
5247
5248        # DEBUG
5249        log.debug("Start annotation with snpeff databases")
5250
5251        # Threads
5252        if not threads:
5253            threads = self.get_threads()
5254        log.debug("Threads: " + str(threads))
5255
5256        # DEBUG
5257        delete_tmp = True
5258        if self.get_config().get("verbosity", "warning") in ["debug"]:
5259            delete_tmp = False
5260            log.debug("Delete tmp files/folders: " + str(delete_tmp))
5261
5262        # Config
5263        config = self.get_config()
5264        log.debug("Config: " + str(config))
5265
5266        # Config - Folders - Databases
5267        databases_folders = (
5268            config.get("folders", {}).get("databases", {}).get("snpeff", ["."])
5269        )
5270        log.debug("Databases annotations: " + str(databases_folders))
5271
5272        # Config - snpEff bin command
5273        snpeff_bin_command = get_bin_command(
5274            bin="snpEff.jar",
5275            tool="snpeff",
5276            bin_type="jar",
5277            config=config,
5278            default_folder=f"{DEFAULT_TOOLS_FOLDER}/snpeff",
5279        )
5280        if not snpeff_bin_command:
5281            msg_err = f"Annotation failed: no snpeff bin '{snpeff_bin_command}'"
5282            log.error(msg_err)
5283            raise ValueError(msg_err)
5284
5285        # Config - snpEff databases
5286        snpeff_databases = (
5287            config.get("folders", {})
5288            .get("databases", {})
5289            .get("snpeff", DEFAULT_SNPEFF_FOLDER)
5290        )
5291        snpeff_databases = full_path(snpeff_databases)
5292        if snpeff_databases is not None and snpeff_databases != "":
5293            log.debug(f"Create snpEff databases folder")
5294            if not os.path.exists(snpeff_databases):
5295                os.makedirs(snpeff_databases)
5296
5297        # Param
5298        param = self.get_param()
5299        log.debug("Param: " + str(param))
5300
5301        # Param
5302        options = param.get("annotation", {}).get("snpeff", {}).get("options", None)
5303        log.debug("Options: " + str(options))
5304
5305        # Param - Assembly
5306        assembly = param.get("assembly", config.get("assembly", DEFAULT_ASSEMBLY))
5307
5308        # Param - Options
5309        snpeff_options = (
5310            param.get("annotation", {}).get("snpeff", {}).get("options", "")
5311        )
5312        snpeff_stats = param.get("annotation", {}).get("snpeff", {}).get("stats", None)
5313        snpeff_csvstats = (
5314            param.get("annotation", {}).get("snpeff", {}).get("csvStats", None)
5315        )
5316        if snpeff_stats:
5317            snpeff_stats = snpeff_stats.replace("OUTPUT", self.get_output())
5318            snpeff_stats = full_path(snpeff_stats)
5319            snpeff_options += f" -stats {snpeff_stats}"
5320        if snpeff_csvstats:
5321            snpeff_csvstats = snpeff_csvstats.replace("OUTPUT", self.get_output())
5322            snpeff_csvstats = full_path(snpeff_csvstats)
5323            snpeff_options += f" -csvStats {snpeff_csvstats}"
5324
5325        # Data
5326        table_variants = self.get_table_variants()
5327
5328        # Check if not empty
5329        log.debug("Check if not empty")
5330        sql_query_chromosomes = (
5331            f"""SELECT count(*) as count FROM {table_variants} as table_variants"""
5332        )
5333        # if not self.conn.execute(f"{sql_query_chromosomes}").df()["count"][0]:
5334        if not self.get_query_to_df(f"{sql_query_chromosomes}")["count"][0]:
5335            log.info(f"VCF empty")
5336            return
5337
5338        # Export in VCF
5339        log.debug("Create initial file to annotate")
5340        tmp_vcf = NamedTemporaryFile(
5341            prefix=self.get_prefix(),
5342            dir=self.get_tmp_dir(),
5343            suffix=".vcf.gz",
5344            delete=True,
5345        )
5346        tmp_vcf_name = tmp_vcf.name
5347
5348        # VCF header
5349        vcf_reader = self.get_header()
5350        log.debug("Initial header: " + str(vcf_reader.infos))
5351
5352        # Existing annotations
5353        for vcf_annotation in self.get_header().infos:
5354
5355            vcf_annotation_line = self.get_header().infos.get(vcf_annotation)
5356            log.debug(
5357                f"Existing annotations in VCF: {vcf_annotation} [{vcf_annotation_line}]"
5358            )
5359
5360        # Memory limit
5361        # if config.get("memory", None):
5362        #     memory_limit = config.get("memory", "8G")
5363        # else:
5364        #     memory_limit = "8G"
5365        memory_limit = self.get_memory("8G")
5366        log.debug(f"memory_limit: {memory_limit}")
5367
5368        # snpEff java options
5369        snpeff_java_options = (
5370            f" -Xmx{memory_limit} -XX:+UseParallelGC -XX:ParallelGCThreads={threads} "
5371        )
5372        log.debug(f"Exomiser java options: {snpeff_java_options}")
5373
5374        force_update_annotation = True
5375
5376        if "ANN" not in self.get_header().infos or force_update_annotation:
5377
5378            # Check snpEff database
5379            log.debug(f"Check snpEff databases {[assembly]}")
5380            databases_download_snpeff(
5381                folder=snpeff_databases, assemblies=[assembly], config=config
5382            )
5383
5384            # Export VCF file
5385            self.export_variant_vcf(
5386                vcf_file=tmp_vcf_name,
5387                remove_info=True,
5388                add_samples=False,
5389                index=True,
5390            )
5391
5392            # Tmp file
5393            err_files = []
5394            tmp_annotate_vcf = NamedTemporaryFile(
5395                prefix=self.get_prefix(),
5396                dir=self.get_tmp_dir(),
5397                suffix=".vcf",
5398                delete=False,
5399            )
5400            tmp_annotate_vcf_name = tmp_annotate_vcf.name
5401            tmp_annotate_vcf_name_err = tmp_annotate_vcf_name + ".err"
5402            err_files.append(tmp_annotate_vcf_name_err)
5403
5404            # Command
5405            snpeff_command = f"{snpeff_bin_command} {assembly} -dataDir {snpeff_databases} {snpeff_options} {tmp_vcf_name} 1>{tmp_annotate_vcf_name} 2>>{tmp_annotate_vcf_name_err}"
5406            log.debug(f"Annotation - snpEff command: {snpeff_command}")
5407            run_parallel_commands([snpeff_command], 1)
5408
5409            # Error messages
5410            log.info(f"Error/Warning messages:")
5411            error_message_command_all = []
5412            error_message_command_warning = []
5413            error_message_command_err = []
5414            for err_file in err_files:
5415                with open(err_file, "r") as f:
5416                    for line in f:
5417                        message = line.strip()
5418                        error_message_command_all.append(message)
5419                        if line.startswith("[W::"):
5420                            error_message_command_warning.append(message)
5421                        if line.startswith("[E::"):
5422                            error_message_command_err.append(f"{err_file}: " + message)
5423            # log info
5424            for message in list(
5425                set(error_message_command_err + error_message_command_warning)
5426            ):
5427                log.info(f"   {message}")
5428            # debug info
5429            for message in list(set(error_message_command_all)):
5430                log.debug(f"   {message}")
5431            # failed
5432            if len(error_message_command_err):
5433                log.error("Annotation failed: Error in commands")
5434                raise ValueError("Annotation failed: Error in commands")
5435
5436            # Find annotation in header
5437            with open(tmp_annotate_vcf_name, "rt") as f:
5438                header_list = self.read_vcf_header(f)
5439            annovar_vcf_header = vcf.Reader(io.StringIO("\n".join(header_list)))
5440
5441            for ann in annovar_vcf_header.infos:
5442                if ann not in self.get_header().infos:
5443                    vcf_reader.infos[ann] = annovar_vcf_header.infos.get(ann)
5444
5445            # Update variants
5446            log.info(f"Annotation - Updating...")
5447            self.update_from_vcf(tmp_annotate_vcf_name)
5448
5449        else:
5450            if "ANN" in self.get_header().infos:
5451                log.debug(f"Existing snpEff annotations in VCF")
5452            if force_update_annotation:
5453                log.debug(f"Existing snpEff annotations in VCF - annotation forced")

This function annotate with snpEff

Parameters

threads: The number of threads to use

Returns

the value of the variable "return_value".

def annotation_annovar(self, threads: int = None) -> None: View Source

5455    def annotation_annovar(self, threads: int = None) -> None:
5456        """
5457        It takes a VCF file, annotates it with Annovar, and then updates the database with the new
5458        annotations
5459
5460        :param threads: number of threads to use
5461        :return: the value of the variable "return_value".
5462        """
5463
5464        # DEBUG
5465        log.debug("Start annotation with Annovar databases")
5466
5467        # Threads
5468        if not threads:
5469            threads = self.get_threads()
5470        log.debug("Threads: " + str(threads))
5471
5472        # Tmp en Err files
5473        tmp_files = []
5474        err_files = []
5475
5476        # DEBUG
5477        delete_tmp = True
5478        if self.get_config().get("verbosity", "warning") in ["debug"]:
5479            delete_tmp = False
5480            log.debug("Delete tmp files/folders: " + str(delete_tmp))
5481
5482        # Config
5483        config = self.get_config()
5484        log.debug("Config: " + str(config))
5485
5486        # Config - Folders - Databases
5487        databases_folders = (
5488            config.get("folders", {}).get("databases", {}).get("annovar", ["."])
5489        )
5490        log.debug("Databases annotations: " + str(databases_folders))
5491
5492        # Config - annovar bin command
5493        annovar_bin_command = get_bin_command(
5494            bin="table_annovar.pl",
5495            tool="annovar",
5496            bin_type="perl",
5497            config=config,
5498            default_folder=f"{DEFAULT_TOOLS_FOLDER}/annovar",
5499        )
5500        if not annovar_bin_command:
5501            msg_err = f"Annotation failed: no annovar bin '{annovar_bin_command}'"
5502            log.error(msg_err)
5503            raise ValueError(msg_err)
5504
5505        # Config - BCFTools bin command
5506        bcftools_bin_command = get_bin_command(
5507            bin="bcftools",
5508            tool="bcftools",
5509            bin_type="bin",
5510            config=config,
5511            default_folder=f"{DEFAULT_TOOLS_FOLDER}/bcftools",
5512        )
5513        if not bcftools_bin_command:
5514            msg_err = f"Annotation failed: no bcftools bin '{bcftools_bin_command}'"
5515            log.error(msg_err)
5516            raise ValueError(msg_err)
5517
5518        # Config - annovar databases
5519        annovar_databases = (
5520            config.get("folders", {})
5521            .get("databases", {})
5522            .get("annovar", DEFAULT_ANNOVAR_FOLDER)
5523        )
5524        if annovar_databases is not None:
5525            if isinstance(annovar_databases, list):
5526                annovar_databases = full_path(annovar_databases[0])
5527                log.warning(f"Annovar databases folder '{annovar_databases}' selected")
5528            annovar_databases = full_path(annovar_databases)
5529            if not os.path.exists(annovar_databases):
5530                log.info(f"Annovar databases folder '{annovar_databases}' created")
5531                Path(annovar_databases).mkdir(parents=True, exist_ok=True)
5532        else:
5533            msg_err = f"Annovar databases configuration failed"
5534            log.error(msg_err)
5535            raise ValueError(msg_err)
5536
5537        # Param
5538        param = self.get_param()
5539        log.debug("Param: " + str(param))
5540
5541        # Param - options
5542        options = param.get("annotation", {}).get("annovar", {}).get("options", {})
5543        log.debug("Options: " + str(options))
5544
5545        # Param - annotations
5546        annotations = (
5547            param.get("annotation", {}).get("annovar", {}).get("annotations", {})
5548        )
5549        log.debug("Annotations: " + str(annotations))
5550
5551        # Param - Assembly
5552        assembly = param.get("assembly", config.get("assembly", DEFAULT_ASSEMBLY))
5553
5554        # Annovar database assembly
5555        annovar_databases_assembly = f"{annovar_databases}/{assembly}"
5556        if annovar_databases_assembly != "" and not os.path.exists(
5557            annovar_databases_assembly
5558        ):
5559            os.makedirs(annovar_databases_assembly)
5560
5561        # Data
5562        table_variants = self.get_table_variants()
5563
5564        # Check if not empty
5565        log.debug("Check if not empty")
5566        sql_query_chromosomes = (
5567            f"""SELECT count(*) as count FROM {table_variants} as table_variants"""
5568        )
5569        sql_query_chromosomes_df = self.get_query_to_df(sql_query_chromosomes)
5570        if not sql_query_chromosomes_df["count"][0]:
5571            log.info(f"VCF empty")
5572            return
5573
5574        # VCF header
5575        vcf_reader = self.get_header()
5576        log.debug("Initial header: " + str(vcf_reader.infos))
5577
5578        # Existing annotations
5579        for vcf_annotation in self.get_header().infos:
5580
5581            vcf_annotation_line = self.get_header().infos.get(vcf_annotation)
5582            log.debug(
5583                f"Existing annotations in VCF: {vcf_annotation} [{vcf_annotation_line}]"
5584            )
5585
5586        force_update_annotation = True
5587
5588        if annotations:
5589
5590            commands = []
5591            tmp_annotates_vcf_name_list = []
5592
5593            # Export in VCF
5594            log.debug("Create initial file to annotate")
5595            tmp_vcf = NamedTemporaryFile(
5596                prefix=self.get_prefix(),
5597                dir=self.get_tmp_dir(),
5598                suffix=".vcf.gz",
5599                delete=False,
5600            )
5601            tmp_vcf_name = tmp_vcf.name
5602            tmp_files.append(tmp_vcf_name)
5603            tmp_files.append(tmp_vcf_name + ".tbi")
5604
5605            # Export VCF file
5606            self.export_variant_vcf(
5607                vcf_file=tmp_vcf_name,
5608                remove_info=".",
5609                add_samples=False,
5610                index=True,
5611            )
5612
5613            # Create file for field rename
5614            log.debug("Create file for field rename")
5615            tmp_rename = NamedTemporaryFile(
5616                prefix=self.get_prefix(),
5617                dir=self.get_tmp_dir(),
5618                suffix=".rename",
5619                delete=False,
5620            )
5621            tmp_rename_name = tmp_rename.name
5622            tmp_files.append(tmp_rename_name)
5623
5624            # Check Annovar database
5625            log.debug(
5626                f"Check Annovar databases {[assembly]}: {list(annotations.keys())}"
5627            )
5628            databases_download_annovar(
5629                folder=annovar_databases,
5630                files=list(annotations.keys()),
5631                assemblies=[assembly],
5632            )
5633
5634            for annotation in annotations:
5635                annotation_fields = annotations[annotation]
5636
5637                if not annotation_fields:
5638                    annotation_fields = {"INFO": None}
5639
5640                log.info(f"Annotations Annovar - database '{annotation}'")
5641                log.debug(f"Annotation '{annotation}' - fields: {annotation_fields}")
5642
5643                # Tmp file for annovar
5644                err_files = []
5645                tmp_annotate_vcf_directory = TemporaryDirectory(
5646                    prefix=self.get_prefix(), dir=self.get_tmp_dir(), suffix=".annovar"
5647                )
5648                tmp_annotate_vcf_prefix = tmp_annotate_vcf_directory.name + "/annovar"
5649                tmp_annotate_vcf_name_annovar = (
5650                    tmp_annotate_vcf_prefix + "." + assembly + "_multianno.vcf"
5651                )
5652                tmp_annotate_vcf_name_err = tmp_annotate_vcf_directory.name + "/.err"
5653                err_files.append(tmp_annotate_vcf_name_err)
5654                tmp_files.append(tmp_annotate_vcf_name_err)
5655
5656                # Tmp file final vcf annotated by annovar
5657                tmp_annotate_vcf = NamedTemporaryFile(
5658                    prefix=self.get_prefix(),
5659                    dir=self.get_tmp_dir(),
5660                    suffix=".vcf.gz",
5661                    delete=False,
5662                )
5663                tmp_annotate_vcf_name = tmp_annotate_vcf.name
5664                tmp_annotates_vcf_name_list.append(tmp_annotate_vcf_name)
5665                tmp_files.append(tmp_annotate_vcf_name)
5666                tmp_files.append(tmp_annotate_vcf_name + ".tbi")
5667
5668                # Number of fields
5669                annotation_list = []
5670                annotation_renamed_list = []
5671
5672                for annotation_field in annotation_fields:
5673
5674                    # field new name, if parametered SKIPPED !!!!!! not managed actually TODO
5675                    annotation_fields_new_name = annotation_fields.get(
5676                        annotation_field, annotation_field
5677                    )
5678                    if not annotation_fields_new_name:
5679                        annotation_fields_new_name = annotation_field
5680
5681                    if (
5682                        force_update_annotation
5683                        or annotation_fields_new_name not in self.get_header().infos
5684                    ):
5685                        annotation_list.append(annotation_field)
5686                        annotation_renamed_list.append(annotation_fields_new_name)
5687                    else:  # annotation_fields_new_name in self.get_header().infos and not force_update_annotation:
5688                        log.warning(
5689                            f"Annotation '{annotation}' - '{annotation_fields_new_name}' - already exists (skipped)"
5690                        )
5691
5692                    # Add rename info
5693                    run_parallel_commands(
5694                        [
5695                            f"echo 'INFO/{annotation_field} {annotation_fields_new_name}' >> {tmp_rename_name}"
5696                        ],
5697                        1,
5698                    )
5699
5700                # log.debug("fields_to_removed: " + str(fields_to_removed))
5701                log.debug("annotation_list: " + str(annotation_list))
5702
5703                # protocol
5704                protocol = annotation
5705
5706                # argument
5707                argument = ""
5708
5709                # operation
5710                operation = "f"
5711                if annotation in ["refGene", "refGeneWithVer"] or annotation.startswith(
5712                    "ensGene"
5713                ):
5714                    operation = "g"
5715                    if options.get("genebase", None):
5716                        argument = f"""'{options.get("genebase","")}'"""
5717                elif annotation in ["cytoBand"]:
5718                    operation = "r"
5719
5720                # argument option
5721                argument_option = ""
5722                if argument != "":
5723                    argument_option = " --argument " + argument
5724
5725                # command options
5726                command_options = f""" --nastring . --vcfinput --polish --dot2underline --thread {threads} """  # --intronhgvs 10
5727                for option in options:
5728                    if option not in ["genebase"]:
5729                        command_options += f""" --{option}={options[option]}"""
5730
5731                # Command
5732
5733                # Command - Annovar
5734                command_annovar = f"""{annovar_bin_command} {tmp_vcf_name} {annovar_databases_assembly} --buildver {assembly} --outfile {tmp_annotate_vcf_prefix} --remove --protocol {protocol} --operation {operation} {argument_option} {command_options} 2>>{tmp_annotate_vcf_name_err} && mv {tmp_annotate_vcf_name_annovar} {tmp_annotate_vcf_name}.tmp.vcf """
5735                tmp_files.append(f"{tmp_annotate_vcf_name}.tmp.vcf")
5736
5737                # Command - start pipe
5738                command_annovar += f""" && {bcftools_bin_command} view --threads={threads} {tmp_annotate_vcf_name}.tmp.vcf 2>>{tmp_annotate_vcf_name_err} """
5739
5740                # Command - Clean INFO/ANNOVAR_DATE (due to Annovar issue with multiple TAGS!)
5741                command_annovar += """ | sed "s/ANNOVAR_DATE=[^;\t]*;//gi" """
5742
5743                # Command - Special characters (refGene annotation)
5744                command_annovar += """ | sed "s/\\\\\\x3b/,/gi" """
5745
5746                # Command - Clean empty fields (with value ".")
5747                command_annovar += """ | awk -F'\\t' -v OFS='\\t' '{if ($0 ~ /^#/) print; else {split($8,a,";");for(i=1;i<=length(a);i++) {split(a[i],b,"=");if(b[2]!=".") {c[b[1]]=b[2]}}; split($8,d,";");for(i=1;i<=length(d);i++) {split(d[i],e,"=");if(c[e[1]]!="") {if(info!="") {info=info";"}; info=info""e[1]"="c[e[1]]}}; if(info!="") {$8=info} else {$8=""}; delete c; info=""; print}}' """
5748
5749                # Command - Extract only needed fields, and remove ANNOVAR fields, and compress and index final file
5750                annovar_fields_to_keep = ["INFO/ANNOVAR_DATE", "INFO/ALLELE_END"]
5751                if "ALL" not in annotation_list and "INFO" not in annotation_list:
5752                    # for ann in annotation_renamed_list:
5753                    for ann in annotation_list:
5754                        annovar_fields_to_keep.append(f"^INFO/{ann}")
5755
5756                command_annovar += f""" | {bcftools_bin_command} annotate --pair-logic exact --threads={threads} -x {",".join(annovar_fields_to_keep)} --rename-annots={tmp_rename_name} -o {tmp_annotate_vcf_name} -Oz 2>>{tmp_annotate_vcf_name_err} """
5757
5758                # Command - indexing
5759                command_annovar += f"""  && tabix {tmp_annotate_vcf_name} """
5760
5761                log.debug(f"Annotation - Annovar command: {command_annovar}")
5762                run_parallel_commands([command_annovar], 1)
5763
5764                # Error messages
5765                log.info(f"Error/Warning messages:")
5766                error_message_command_all = []
5767                error_message_command_warning = []
5768                error_message_command_err = []
5769                for err_file in err_files:
5770                    with open(err_file, "r") as f:
5771                        for line in f:
5772                            message = line.strip()
5773                            error_message_command_all.append(message)
5774                            if line.startswith("[W::") or line.startswith("WARNING"):
5775                                error_message_command_warning.append(message)
5776                            if line.startswith("[E::") or line.startswith("ERROR"):
5777                                error_message_command_err.append(
5778                                    f"{err_file}: " + message
5779                                )
5780                # log info
5781                for message in list(
5782                    set(error_message_command_err + error_message_command_warning)
5783                ):
5784                    log.info(f"   {message}")
5785                # debug info
5786                for message in list(set(error_message_command_all)):
5787                    log.debug(f"   {message}")
5788                # failed
5789                if len(error_message_command_err):
5790                    log.error("Annotation failed: Error in commands")
5791                    raise ValueError("Annotation failed: Error in commands")
5792
5793            if tmp_annotates_vcf_name_list:
5794
5795                # List of annotated files
5796                tmp_annotates_vcf_name_to_merge = " ".join(tmp_annotates_vcf_name_list)
5797
5798                # Tmp file
5799                tmp_annotate_vcf = NamedTemporaryFile(
5800                    prefix=self.get_prefix(),
5801                    dir=self.get_tmp_dir(),
5802                    suffix=".vcf.gz",
5803                    delete=False,
5804                )
5805                tmp_annotate_vcf_name = tmp_annotate_vcf.name
5806                tmp_files.append(tmp_annotate_vcf_name)
5807                tmp_annotate_vcf_name_err = tmp_annotate_vcf_name + ".err"
5808                err_files.append(tmp_annotate_vcf_name_err)
5809                tmp_files.append(tmp_annotate_vcf_name_err)
5810
5811                # Command merge
5812                merge_command = f"{bcftools_bin_command} merge --force-samples --threads={threads} {tmp_vcf_name} {tmp_annotates_vcf_name_to_merge} -o {tmp_annotate_vcf_name} -Oz 2>>{tmp_annotate_vcf_name_err} "
5813                log.info(
5814                    f"Annotation Annovar - Annotation merging "
5815                    + str(len(tmp_annotates_vcf_name_list))
5816                    + " annotated files"
5817                )
5818                log.debug(f"Annotation - merge command: {merge_command}")
5819                run_parallel_commands([merge_command], 1)
5820
5821                # Find annotation in header
5822                with bgzf.open(tmp_annotate_vcf_name, "rt") as f:
5823                    header_list = self.read_vcf_header(f)
5824                annovar_vcf_header = vcf.Reader(io.StringIO("\n".join(header_list)))
5825
5826                for ann in annovar_vcf_header.infos:
5827                    if ann not in self.get_header().infos:
5828                        vcf_reader.infos[ann] = annovar_vcf_header.infos.get(ann)
5829
5830                # Update variants
5831                log.info(f"Annotation Annovar - Updating...")
5832                self.update_from_vcf(tmp_annotate_vcf_name)
5833
5834            # Clean files
5835            # Tmp file remove command
5836            if True:
5837                tmp_files_remove_command = ""
5838                if tmp_files:
5839                    tmp_files_remove_command = " ".join(tmp_files)
5840                clean_command = f" rm -f {tmp_files_remove_command} "
5841                log.debug(f"Annotation Annovar - Annotation cleaning ")
5842                log.debug(f"Annotation - cleaning command: {clean_command}")
5843                run_parallel_commands([clean_command], 1)

It takes a VCF file, annotates it with Annovar, and then updates the database with the new annotations

Parameters

threads: number of threads to use

Returns

the value of the variable "return_value".

def annotation_parquet(self, threads: int = None) -> None: View Source

5846    def annotation_parquet(self, threads: int = None) -> None:
5847        """
5848        It takes a VCF file, and annotates it with a parquet file
5849
5850        :param threads: number of threads to use for the annotation
5851        :return: the value of the variable "result".
5852        """
5853
5854        # DEBUG
5855        log.debug("Start annotation with parquet databases")
5856
5857        # Threads
5858        if not threads:
5859            threads = self.get_threads()
5860        log.debug("Threads: " + str(threads))
5861
5862        # DEBUG
5863        delete_tmp = True
5864        if self.get_config().get("verbosity", "warning") in ["debug"]:
5865            delete_tmp = False
5866            log.debug("Delete tmp files/folders: " + str(delete_tmp))
5867
5868        # Config
5869        databases_folders = set(
5870            self.get_config()
5871            .get("folders", {})
5872            .get("databases", {})
5873            .get("annotations", ["."])
5874            + self.get_config()
5875            .get("folders", {})
5876            .get("databases", {})
5877            .get("parquet", ["."])
5878        )
5879        log.debug("Databases annotations: " + str(databases_folders))
5880
5881        # Param
5882        annotations = (
5883            self.get_param()
5884            .get("annotation", {})
5885            .get("parquet", {})
5886            .get("annotations", None)
5887        )
5888        log.debug("Annotations: " + str(annotations))
5889
5890        # Assembly
5891        assembly = self.get_param().get(
5892            "assembly", self.get_config().get("assembly", DEFAULT_ASSEMBLY)
5893        )
5894
5895        # Force Update Annotation
5896        force_update_annotation = (
5897            self.get_param()
5898            .get("annotation", {})
5899            .get("options", {})
5900            .get("annotations_update", False)
5901        )
5902        log.debug(f"force_update_annotation={force_update_annotation}")
5903        force_append_annotation = (
5904            self.get_param()
5905            .get("annotation", {})
5906            .get("options", {})
5907            .get("annotations_append", False)
5908        )
5909        log.debug(f"force_append_annotation={force_append_annotation}")
5910
5911        # Data
5912        table_variants = self.get_table_variants()
5913
5914        # Check if not empty
5915        log.debug("Check if not empty")
5916        sql_query_chromosomes_df = self.get_query_to_df(
5917            f"""SELECT count(*) as count FROM {table_variants} as table_variants LIMIT 1"""
5918        )
5919        if not sql_query_chromosomes_df["count"][0]:
5920            log.info(f"VCF empty")
5921            return
5922
5923        # VCF header
5924        vcf_reader = self.get_header()
5925        log.debug("Initial header: " + str(vcf_reader.infos))
5926
5927        # Nb Variants POS
5928        log.debug("NB Variants Start")
5929        nb_variants = self.conn.execute(
5930            f"SELECT count(*) AS count FROM variants"
5931        ).fetchdf()["count"][0]
5932        log.debug("NB Variants Stop")
5933
5934        # Existing annotations
5935        for vcf_annotation in self.get_header().infos:
5936
5937            vcf_annotation_line = self.get_header().infos.get(vcf_annotation)
5938            log.debug(
5939                f"Existing annotations in VCF: {vcf_annotation} [{vcf_annotation_line}]"
5940            )
5941
5942        # Added columns
5943        added_columns = []
5944
5945        # drop indexes
5946        log.debug(f"Drop indexes...")
5947        self.drop_indexes()
5948
5949        if annotations:
5950
5951            if "ALL" in annotations:
5952
5953                all_param = annotations.get("ALL", {})
5954                all_param_formats = all_param.get("formats", None)
5955                all_param_releases = all_param.get("releases", None)
5956
5957                databases_infos_dict = self.scan_databases(
5958                    database_formats=all_param_formats,
5959                    database_releases=all_param_releases,
5960                )
5961                for database_infos in databases_infos_dict.keys():
5962                    if database_infos not in annotations:
5963                        annotations[database_infos] = {"INFO": None}
5964
5965            for annotation in annotations:
5966
5967                if annotation in ["ALL"]:
5968                    continue
5969
5970                # Annotation Name
5971                annotation_name = os.path.basename(annotation)
5972
5973                # Annotation fields
5974                annotation_fields = annotations[annotation]
5975                if not annotation_fields:
5976                    annotation_fields = {"INFO": None}
5977
5978                log.debug(f"Annotation '{annotation_name}'")
5979                log.debug(
5980                    f"Annotation '{annotation_name}' - fields: {annotation_fields}"
5981                )
5982
5983                # Create Database
5984                database = Database(
5985                    database=annotation,
5986                    databases_folders=databases_folders,
5987                    assembly=assembly,
5988                )
5989
5990                # Find files
5991                parquet_file = database.get_database()
5992                parquet_hdr_file = database.get_header_file()
5993                parquet_type = database.get_type()
5994
5995                # Check if files exists
5996                if not parquet_file or not parquet_hdr_file:
5997                    msg_err_list = []
5998                    if not parquet_file:
5999                        msg_err_list.append(
6000                            f"Annotation failed: Annotation file not found"
6001                        )
6002                    if parquet_file and not parquet_hdr_file:
6003                        msg_err_list.append(
6004                            f"Annotation failed: Annotation file '{parquet_file}' header not found. Check for file '{parquet_file}.hdr'"
6005                        )
6006
6007                    log.error(". ".join(msg_err_list))
6008                    raise ValueError(". ".join(msg_err_list))
6009                else:
6010                    # Get parquet connexion
6011                    parquet_sql_attach = database.get_sql_database_attach(
6012                        output="query"
6013                    )
6014                    if parquet_sql_attach:
6015                        self.conn.execute(parquet_sql_attach)
6016                    parquet_file_link = database.get_sql_database_link()
6017                    # Log
6018                    log.debug(
6019                        f"Annotation '{annotation_name}' - file: "
6020                        + str(parquet_file)
6021                        + " and "
6022                        + str(parquet_hdr_file)
6023                    )
6024
6025                    # Database full header columns
6026                    parquet_hdr_vcf_header_columns = database.get_header_file_columns(
6027                        parquet_hdr_file
6028                    )
6029                    # Log
6030                    log.debug(
6031                        "Annotation database header columns : "
6032                        + str(parquet_hdr_vcf_header_columns)
6033                    )
6034
6035                    # Load header as VCF object
6036                    parquet_hdr_vcf_header_infos = database.get_header().infos
6037                    # Log
6038                    log.debug(
6039                        "Annotation database header: "
6040                        + str(parquet_hdr_vcf_header_infos)
6041                    )
6042
6043                    # Get extra infos
6044                    parquet_columns = database.get_extra_columns()
6045                    # Log
6046                    log.debug("Annotation database Columns: " + str(parquet_columns))
6047
6048                    # Add extra columns if "ALL" in annotation_fields
6049                    # if "ALL" in annotation_fields:
6050                    #     allow_add_extra_column = True
6051                    if "ALL" in annotation_fields and database.get_extra_columns():
6052                        for extra_column in database.get_extra_columns():
6053                            if (
6054                                extra_column not in annotation_fields
6055                                and extra_column.replace("INFO/", "")
6056                                not in parquet_hdr_vcf_header_infos
6057                            ):
6058                                parquet_hdr_vcf_header_infos[extra_column] = (
6059                                    vcf.parser._Info(
6060                                        extra_column,
6061                                        ".",
6062                                        "String",
6063                                        f"{extra_column} description",
6064                                        "unknown",
6065                                        "unknown",
6066                                        self.code_type_map["String"],
6067                                    )
6068                                )
6069
6070                    # For all fields in database
6071                    annotation_fields_all = False
6072                    if "ALL" in annotation_fields or "INFO" in annotation_fields:
6073                        annotation_fields_all = True
6074                        annotation_fields = {
6075                            key: key for key in parquet_hdr_vcf_header_infos
6076                        }
6077
6078                        log.debug(
6079                            "Annotation database header - All annotations added: "
6080                            + str(annotation_fields)
6081                        )
6082
6083                    # Init
6084
6085                    # List of annotation fields to use
6086                    sql_query_annotation_update_info_sets = []
6087
6088                    # List of annotation to agregate
6089                    sql_query_annotation_to_agregate = []
6090
6091                    # Number of fields
6092                    nb_annotation_field = 0
6093
6094                    # Annotation fields processed
6095                    annotation_fields_processed = []
6096
6097                    # Columns mapping
6098                    map_columns = database.map_columns(
6099                        columns=annotation_fields, prefixes=["INFO/"]
6100                    )
6101
6102                    # Query dict for fields to remove (update option)
6103                    query_dict_remove = {}
6104
6105                    # Fetch Anotation fields
6106                    for annotation_field in annotation_fields:
6107
6108                        # annotation_field_column
6109                        annotation_field_column = map_columns.get(
6110                            annotation_field, "INFO"
6111                        )
6112
6113                        # field new name, if parametered
6114                        annotation_fields_new_name = annotation_fields.get(
6115                            annotation_field, annotation_field
6116                        )
6117                        if not annotation_fields_new_name:
6118                            annotation_fields_new_name = annotation_field
6119
6120                        # To annotate
6121                        # force_update_annotation = True
6122                        # force_append_annotation = True
6123                        # if annotation_field in parquet_hdr_vcf_header_infos and (force_update_annotation or (annotation_fields_new_name not in self.get_header().infos)):
6124                        if annotation_field in parquet_hdr_vcf_header_infos and (
6125                            force_update_annotation
6126                            or force_append_annotation
6127                            or (
6128                                annotation_fields_new_name
6129                                not in self.get_header().infos
6130                            )
6131                        ):
6132
6133                            # Add field to annotation to process list
6134                            annotation_fields_processed.append(
6135                                annotation_fields_new_name
6136                            )
6137
6138                            # explode infos for the field
6139                            annotation_fields_new_name_info_msg = ""
6140                            if (
6141                                force_update_annotation
6142                                and annotation_fields_new_name
6143                                in self.get_header().infos
6144                            ):
6145                                # Remove field from INFO
6146                                query = f"""
6147                                    UPDATE {table_variants} as table_variants
6148                                    SET INFO = REGEXP_REPLACE(
6149                                                concat(table_variants.INFO,''),
6150                                                ';*{annotation_fields_new_name}=[^;]*',
6151                                                ''
6152                                                )
6153                                    WHERE concat(';',table_variants.INFO) LIKE '%;{annotation_fields_new_name}=%'
6154                                """
6155                                annotation_fields_new_name_info_msg = " [update]"
6156                                query_dict_remove[
6157                                    f"remove 'INFO/{annotation_fields_new_name}'"
6158                                ] = query
6159
6160                            # Sep between fields in INFO
6161                            nb_annotation_field += 1
6162                            if nb_annotation_field > 1:
6163                                annotation_field_sep = ";"
6164                            else:
6165                                annotation_field_sep = ""
6166
6167                            log.info(
6168                                f"Annotation '{annotation_name}' - '{annotation_field}' -> '{annotation_fields_new_name}'{annotation_fields_new_name_info_msg}"
6169                            )
6170
6171                            # Add INFO field to header
6172                            parquet_hdr_vcf_header_infos_number = (
6173                                parquet_hdr_vcf_header_infos[annotation_field].num
6174                                or "."
6175                            )
6176                            parquet_hdr_vcf_header_infos_type = (
6177                                parquet_hdr_vcf_header_infos[annotation_field].type
6178                                or "String"
6179                            )
6180                            parquet_hdr_vcf_header_infos_description = (
6181                                parquet_hdr_vcf_header_infos[annotation_field].desc
6182                                or f"{annotation_field} description"
6183                            )
6184                            parquet_hdr_vcf_header_infos_source = (
6185                                parquet_hdr_vcf_header_infos[annotation_field].source
6186                                or "unknown"
6187                            )
6188                            parquet_hdr_vcf_header_infos_version = (
6189                                parquet_hdr_vcf_header_infos[annotation_field].version
6190                                or "unknown"
6191                            )
6192
6193                            vcf_reader.infos[annotation_fields_new_name] = (
6194                                vcf.parser._Info(
6195                                    annotation_fields_new_name,
6196                                    parquet_hdr_vcf_header_infos_number,
6197                                    parquet_hdr_vcf_header_infos_type,
6198                                    parquet_hdr_vcf_header_infos_description,
6199                                    parquet_hdr_vcf_header_infos_source,
6200                                    parquet_hdr_vcf_header_infos_version,
6201                                    self.code_type_map[
6202                                        parquet_hdr_vcf_header_infos_type
6203                                    ],
6204                                )
6205                            )
6206
6207                            # Append
6208                            if force_append_annotation:
6209                                query_case_when_append = f""" AND REGEXP_EXTRACT(concat(';', table_variants.INFO), ';{annotation_fields_new_name}=([^;]*)',1) IN ('','.') """
6210                            else:
6211                                query_case_when_append = ""
6212
6213                            # Annotation/Update query fields
6214                            # Found in INFO column
6215                            if (
6216                                annotation_field_column == "INFO"
6217                                and "INFO" in parquet_hdr_vcf_header_columns
6218                            ):
6219                                sql_query_annotation_update_info_sets.append(
6220                                    f"""
6221                                CASE WHEN REGEXP_EXTRACT(concat(';', table_parquet.INFO), ';{annotation_field}=([^;]*)',1) NOT IN ('','.') {query_case_when_append}
6222                                        THEN concat('{annotation_field_sep}', '{annotation_fields_new_name}=', REGEXP_EXTRACT(concat(';', table_parquet.INFO), ';{annotation_field}=([^;]*)',1))
6223                                        ELSE ''
6224                                    END
6225                                """
6226                                )
6227                            # Found in a specific column
6228                            else:
6229                                sql_query_annotation_update_info_sets.append(
6230                                    f"""
6231                                CASE WHEN CAST(table_parquet."{annotation_field_column}" AS VARCHAR) NOT IN ('','.') {query_case_when_append}
6232                                        THEN concat('{annotation_field_sep}', '{annotation_fields_new_name}=', replace(CAST(table_parquet."{annotation_field_column}" AS VARCHAR), ';', ','))
6233                                        ELSE ''
6234                                    END
6235                                """
6236                                )
6237                                sql_query_annotation_to_agregate.append(
6238                                    f""" string_agg(table_parquet_from."{annotation_field_column}", ',') AS "{annotation_field_column}" """
6239                                )
6240
6241                        # Not to annotate
6242                        else:
6243
6244                            if force_update_annotation:
6245                                annotation_message = "forced"
6246                            else:
6247                                annotation_message = "skipped"
6248
6249                            if annotation_field not in parquet_hdr_vcf_header_infos:
6250                                log.warning(
6251                                    f"Annotation '{annotation_name}' - '{annotation_field}' [{nb_annotation_field}] - not available in parquet file"
6252                                )
6253                            if annotation_fields_new_name in self.get_header().infos:
6254                                log.warning(
6255                                    f"Annotation '{annotation_name}' - '{annotation_fields_new_name}' [{nb_annotation_field}] - already exists in header ({annotation_message})"
6256                                )
6257
6258                    # Check if ALL fields have to be annotated. Thus concat all INFO field
6259                    # allow_annotation_full_info = True
6260                    allow_annotation_full_info = not force_append_annotation
6261
6262                    if parquet_type in ["regions"]:
6263                        allow_annotation_full_info = False
6264
6265                    if (
6266                        allow_annotation_full_info
6267                        and nb_annotation_field == len(annotation_fields)
6268                        and annotation_fields_all
6269                        and (
6270                            "INFO" in parquet_hdr_vcf_header_columns
6271                            and "INFO" in database.get_extra_columns()
6272                        )
6273                    ):
6274                        log.debug("Column INFO annotation enabled")
6275                        sql_query_annotation_update_info_sets = []
6276                        sql_query_annotation_update_info_sets.append(
6277                            f" table_parquet.INFO "
6278                        )
6279
6280                    if sql_query_annotation_update_info_sets:
6281
6282                        # Annotate
6283                        log.info(f"Annotation '{annotation_name}' - Annotation...")
6284
6285                        # Join query annotation update info sets for SQL
6286                        sql_query_annotation_update_info_sets_sql = ",".join(
6287                            sql_query_annotation_update_info_sets
6288                        )
6289
6290                        # Check chromosomes list (and variants infos)
6291                        sql_query_chromosomes = f"""
6292                            SELECT table_variants."#CHROM" as CHROM, count(*) AS count_variants, min(POS) AS min_variants, MAX(POS) AS max_variants
6293                            FROM {table_variants} as table_variants
6294                            GROUP BY table_variants."#CHROM"
6295                            ORDER BY table_variants."#CHROM"
6296                            """
6297                        sql_query_chromosomes_df = self.conn.execute(
6298                            sql_query_chromosomes
6299                        ).df()
6300                        sql_query_chromosomes_dict = {
6301                            entry["CHROM"]: {
6302                                "count": entry["count_variants"],
6303                                "min": entry["min_variants"],
6304                                "max": entry["max_variants"],
6305                            }
6306                            for index, entry in sql_query_chromosomes_df.iterrows()
6307                        }
6308
6309                        # Init
6310                        nb_of_query = 0
6311                        nb_of_variant_annotated = 0
6312                        query_dict = query_dict_remove
6313
6314                        # for chrom in sql_query_chromosomes_df["CHROM"]:
6315                        for chrom in sql_query_chromosomes_dict:
6316
6317                            # Number of variant by chromosome
6318                            nb_of_variant_by_chrom = sql_query_chromosomes_dict.get(
6319                                chrom, {}
6320                            ).get("count", 0)
6321
6322                            log.debug(
6323                                f"Annotation '{annotation_name}' - Chromosome '{chrom}' [{nb_of_variant_by_chrom} variants]..."
6324                            )
6325
6326                            # Annotation with regions database
6327                            if parquet_type in ["regions"]:
6328                                sql_query_annotation_from_clause = f"""
6329                                    FROM (
6330                                        SELECT 
6331                                            '{chrom}' AS \"#CHROM\",
6332                                            table_variants_from.\"POS\" AS \"POS\",
6333                                            {",".join(sql_query_annotation_to_agregate)}
6334                                        FROM {table_variants} as table_variants_from
6335                                        LEFT JOIN {parquet_file_link} as table_parquet_from ON (
6336                                            table_parquet_from."#CHROM" = '{chrom}'
6337                                            AND table_variants_from.\"POS\" <= table_parquet_from.\"END\"
6338                                            AND table_variants_from.\"POS\" + (len(table_variants_from.\"REF\")-1) >= (table_parquet_from.\"START\"+1)
6339                                        )
6340                                        WHERE table_variants_from.\"#CHROM\" in ('{chrom}')
6341                                        GROUP BY table_variants_from.\"POS\"
6342                                        )
6343                                        as table_parquet
6344                                """
6345
6346                                sql_query_annotation_where_clause = """
6347                                    table_parquet.\"#CHROM\" = table_variants.\"#CHROM\"
6348                                    AND table_parquet.\"POS\" = table_variants.\"POS\"
6349                                """
6350
6351                            # Annotation with variants database
6352                            else:
6353                                sql_query_annotation_from_clause = f"""
6354                                    FROM {parquet_file_link} as table_parquet
6355                                """
6356                                sql_query_annotation_where_clause = f"""
6357                                    table_variants."#CHROM" = '{chrom}'
6358                                    AND table_parquet.\"#CHROM\" = table_variants.\"#CHROM\" 
6359                                    AND table_parquet.\"POS\" = table_variants.\"POS\"
6360                                    AND table_parquet.\"ALT\" = table_variants.\"ALT\"
6361                                    AND table_parquet.\"REF\" = table_variants.\"REF\"
6362                                """
6363
6364                            # Create update query
6365                            sql_query_annotation_chrom_interval_pos = f"""
6366                                UPDATE {table_variants} as table_variants
6367                                    SET INFO = 
6368                                        concat(
6369                                            CASE WHEN table_variants.INFO NOT IN ('','.')
6370                                                THEN table_variants.INFO
6371                                                ELSE ''
6372                                            END
6373                                            ,
6374                                            CASE WHEN table_variants.INFO NOT IN ('','.')
6375                                                        AND (
6376                                                        concat({sql_query_annotation_update_info_sets_sql})
6377                                                        )
6378                                                        NOT IN ('','.') 
6379                                                    THEN ';'
6380                                                    ELSE ''
6381                                            END
6382                                            ,
6383                                            {sql_query_annotation_update_info_sets_sql}
6384                                            )
6385                                    {sql_query_annotation_from_clause}
6386                                    WHERE {sql_query_annotation_where_clause}
6387                                    ;
6388                                """
6389
6390                            # Add update query to dict
6391                            query_dict[
6392                                f"{chrom} [{nb_of_variant_by_chrom} variants]"
6393                            ] = sql_query_annotation_chrom_interval_pos
6394
6395                        nb_of_query = len(query_dict)
6396                        num_query = 0
6397
6398                        # SET max_expression_depth TO x
6399                        self.conn.execute("SET max_expression_depth TO 10000")
6400
6401                        for query_name in query_dict:
6402                            query = query_dict[query_name]
6403                            num_query += 1
6404                            log.info(
6405                                f"Annotation '{annotation_name}' - Annotation - Query [{num_query}/{nb_of_query}] {query_name}..."
6406                            )
6407                            result = self.conn.execute(query)
6408                            nb_of_variant_annotated_by_query = result.df()["Count"][0]
6409                            nb_of_variant_annotated += nb_of_variant_annotated_by_query
6410                            log.info(
6411                                f"Annotation '{annotation_name}' - Annotation - Query [{num_query}/{nb_of_query}] {query_name} - {nb_of_variant_annotated_by_query} variants annotated"
6412                            )
6413
6414                        log.info(
6415                            f"Annotation '{annotation_name}' - Annotation of {nb_of_variant_annotated} variants out of {nb_variants} (with {nb_of_query} queries)"
6416                        )
6417
6418                    else:
6419
6420                        log.info(
6421                            f"Annotation '{annotation_name}' - No Annotations available"
6422                        )
6423
6424                    log.debug("Final header: " + str(vcf_reader.infos))
6425
6426        # Remove added columns
6427        for added_column in added_columns:
6428            self.drop_column(column=added_column)

It takes a VCF file, and annotates it with a parquet file

Parameters

threads: number of threads to use for the annotation

Returns

the value of the variable "result".

def annotation_splice(self, threads: int = None) -> None: View Source

6430    def annotation_splice(self, threads: int = None) -> None:
6431        """
6432        This function annotate with snpEff
6433
6434        :param threads: The number of threads to use
6435        :return: the value of the variable "return_value".
6436        """
6437
6438        # DEBUG
6439        log.debug("Start annotation with splice tools")
6440
6441        # Threads
6442        if not threads:
6443            threads = self.get_threads()
6444        log.debug("Threads: " + str(threads))
6445
6446        # DEBUG
6447        delete_tmp = True
6448        if self.get_config().get("verbosity", "warning") in ["debug"]:
6449            delete_tmp = False
6450            log.debug("Delete tmp files/folders: " + str(delete_tmp))
6451
6452        # Config
6453        config = self.get_config()
6454        log.debug("Config: " + str(config))
6455        splice_config = config.get("tools", {}).get("splice", {})
6456        if not splice_config:
6457            splice_config = DEFAULT_TOOLS_BIN.get("splice", {})
6458            msg_err = "No Splice tool config"
6459            raise ValueError(msg_err)
6460        log.debug(f"splice_config: {splice_config}")
6461
6462        # Config - Folders - Databases
6463        databases_folders = (
6464            config.get("folders", {}).get("databases", {}).get("splice", ["."])
6465        )
6466        log.debug("Databases annotations: " + str(databases_folders))
6467
6468        # Splice docker image
6469        splice_docker_image = splice_config.get("docker").get("image")
6470
6471        # Pull splice image if it's not already there
6472        if not check_docker_image_exists(splice_docker_image):
6473            log.warning(
6474                f"Annotation: splice docker image {splice_docker_image} not found locally, trying to pull from dockerhub"
6475            )
6476            try:
6477                command(f"docker pull {splice_config.get('docker').get('image')}")
6478            except subprocess.CalledProcessError:
6479                msg_err = f"Unable to find docker {splice_docker_image} on dockerhub"
6480                log.error(msg_err)
6481                raise ValueError(msg_err)
6482
6483        # Config - splice databases
6484        splice_databases = (
6485            config.get("folders", {})
6486            .get("databases", {})
6487            .get("splice", DEFAULT_SPLICE_FOLDER)
6488        )
6489        splice_databases = full_path(splice_databases)
6490
6491        # Param
6492        param = self.get_param()
6493        log.debug("Param: " + str(param))
6494
6495        # Param
6496        options = param.get("annotation", {}).get("splice", {}).get("options", {})
6497        log.debug("Options: " + str(options))
6498
6499        # Data
6500        table_variants = self.get_table_variants()
6501
6502        # Check if not empty
6503        log.debug("Check if not empty")
6504        sql_query_chromosomes = (
6505            f"""SELECT count(*) as count FROM {table_variants} as table_variants"""
6506        )
6507        if not self.get_query_to_df(f"{sql_query_chromosomes}")["count"][0]:
6508            log.info("VCF empty")
6509            return None
6510
6511        # Export in VCF
6512        log.debug("Create initial file to annotate")
6513
6514        # Create output folder / work folder
6515        if options.get("output_folder", ""):
6516            output_folder = options.get("output_folder", "")
6517            if not os.path.exists(output_folder):
6518                Path(output_folder).mkdir(parents=True, exist_ok=True)
6519        else:
6520            output_folder = os.path.join(self.get_tmp_dir(), f"splice-{get_random()}")
6521            if not os.path.exists(output_folder):
6522                Path(output_folder).mkdir(parents=True, exist_ok=True)
6523
6524        if options.get("workdir", ""):
6525            workdir = options.get("workdir", "")
6526        else:
6527            workdir = "/work"
6528
6529        # Create tmp VCF file
6530        tmp_vcf = NamedTemporaryFile(
6531            prefix=self.get_prefix(),
6532            dir=output_folder,
6533            suffix=".vcf",
6534            delete=False,
6535        )
6536        tmp_vcf_name = tmp_vcf.name
6537
6538        # VCF header
6539        header = self.get_header()
6540
6541        # Existing annotations
6542        for vcf_annotation in self.get_header().infos:
6543
6544            vcf_annotation_line = self.get_header().infos.get(vcf_annotation)
6545            log.debug(
6546                f"Existing annotations in VCF: {vcf_annotation} [{vcf_annotation_line}]"
6547            )
6548
6549        # Memory limit
6550        if config.get("memory", None):
6551            memory_limit = config.get("memory", "8G").upper()
6552            # upper()
6553        else:
6554            memory_limit = "8G"
6555        log.debug(f"memory_limit: {memory_limit}")
6556
6557        # Check number of variants to annotate
6558        where_clause_regex_spliceai = r"SpliceAI_\w+"
6559        where_clause_regex_spip = r"SPiP_\w+"
6560        where_clause = f""" WHERE NOT regexp_matches("INFO", '{where_clause_regex_spliceai}') AND NOT regexp_matches("INFO", '{where_clause_regex_spip}')"""
6561        df_list_of_variants_to_annotate = self.get_query_to_df(
6562            query=f""" SELECT * FROM variants {where_clause} """
6563        )
6564        if len(df_list_of_variants_to_annotate) == 0:
6565            log.warning(
6566                f"No variants to annotate with splice. Variants probably already annotated with splice"
6567            )
6568            return None
6569        else:
6570            log.info(f"Annotation: {len(df_list_of_variants_to_annotate)} variants")
6571
6572        # Export VCF file
6573        self.export_variant_vcf(
6574            vcf_file=tmp_vcf_name,
6575            remove_info=True,
6576            add_samples=True,
6577            index=False,
6578            where_clause=where_clause,
6579        )
6580        mount = [f" -v {path}:{path}:rw" for path in [output_folder]]
6581        if any(value for value in splice_config.values() if value is None):
6582            log.warning("At least one splice config parameter is empty")
6583            # exit annotation_splice
6584            return None
6585
6586        # Params in splice nf
6587        def check_values(dico: dict):
6588            """
6589            Ensure parameters for NF splice pipeline
6590            """
6591            for key, val in dico.items():
6592                if key == "genome":
6593                    if any(
6594                        assemb in options.get("genome", {})
6595                        for assemb in ["hg19", "GRCh37", "grch37", "GRCH37"]
6596                    ):
6597                        yield f"--{key} hg19"
6598                    elif any(
6599                        assemb in options.get("genome", {})
6600                        for assemb in ["hg38", "GRCh38", "grch38", "GRCH38"]
6601                    ):
6602                        yield f"--{key} hg38"
6603                elif (
6604                    (isinstance(val, str) and val)
6605                    or isinstance(val, int)
6606                    or isinstance(val, bool)
6607                ):
6608                    yield f"--{key} {val}"
6609
6610        # Genome
6611        genome = options.get("genome", config.get("assembly", DEFAULT_ASSEMBLY))
6612        options["genome"] = genome
6613        # NF params
6614        nf_params = []
6615        # Add options
6616        if options:
6617            log.debug(options)
6618            nf_params = list(check_values(options))
6619            log.debug(f"Splice NF params: {' '.join(nf_params)}")
6620        else:
6621            log.debug("No NF params provided")
6622        # Add threads
6623        if "threads" not in options.keys():
6624            nf_params.append(f"--threads {threads}")
6625        # Genome path
6626        genome_path = find_genome(
6627            config.get("folders", {})
6628            .get("databases", {})
6629            .get("genomes", DEFAULT_GENOME_FOLDER),
6630            file=f"{genome}.fa",
6631        )
6632        # Add genome path
6633        if not genome_path:
6634            raise ValueError(
6635                f"Can't find genome assembly {genome}.fa in {config.get('folders', {}).get('databases', {}).get('genomes', DEFAULT_GENOME_FOLDER)}"
6636            )
6637        else:
6638            log.debug(f"Genome: {genome_path}")
6639            nf_params.append(f"--genome_path {genome_path}")
6640
6641        def splice_annotations(options: dict = {}, config: dict = {}) -> list:
6642            """
6643            Setting up updated databases for SPiP and SpliceAI
6644            """
6645
6646            try:
6647
6648                # SpliceAI assembly transcriptome
6649                spliceai_assembly = os.path.join(
6650                    config.get("folders", {}).get("databases", {}).get("spliceai", {}),
6651                    options.get("genome"),
6652                    "transcriptome",
6653                )
6654                spip_assembly = options.get("genome")
6655
6656                spip = find(
6657                    f"transcriptome_{spip_assembly}.RData",
6658                    config.get("folders", {}).get("databases", {}).get("spip", {}),
6659                )
6660                spliceai = find("spliceai.refseq.txt", spliceai_assembly)
6661                log.debug(f"SPiP annotations: {spip}")
6662                log.debug(f"SpliceAI annotations: {spliceai}")
6663                if spip and spliceai:
6664                    return [
6665                        f"--spip_transcriptome {spip}",
6666                        f"--spliceai_transcriptome {spliceai}",
6667                    ]
6668                else:
6669                    log.warning(
6670                        "Can't find splice databases in configuration, use annotations file from image"
6671                    )
6672            except TypeError:
6673                log.warning(
6674                    "Can't find splice databases in configuration, use annotations file from image"
6675                )
6676                return []
6677
6678        # Add options, check if transcriptome option have already beend provided
6679        if (
6680            "spip_transcriptome" not in nf_params
6681            and "spliceai_transcriptome" not in nf_params
6682        ):
6683            splice_reference = splice_annotations(options, config)
6684            if splice_reference:
6685                nf_params.extend(splice_reference)
6686        # nf_params.append(f"--output_folder {output_folder}")
6687        random_uuid = f"HOWARD-SPLICE-{get_random()}"
6688        cmd = f"nextflow -log {os.path.join(output_folder, f'{random_uuid}.log')} -c /app/SpliceToolBox/src/splicetoolbox/nextflow/nextflow.docker.config run /app/SpliceToolBox/src/splicetoolbox/nextflow/main.nf -entry SPLICE --vcf {tmp_vcf_name} {' '.join(nf_params)} -profile standard,conda,singularity,report,timeline"
6689        log.debug(cmd)
6690        splice_config["docker"]["command"] = cmd
6691
6692        # Ensure proxy is set
6693        proxy = [
6694            f"-e {var}={os.getenv(var)}"
6695            for var in ["https_proxy", "http_proxy", "ftp_proxy"]
6696            if os.getenv(var) is not None
6697        ]
6698        docker_cmd = get_bin_command(
6699            tool="splice",
6700            bin_type="docker",
6701            config=config,
6702            default_folder=f"{DEFAULT_TOOLS_FOLDER}/docker",
6703            add_options=f"--name {random_uuid} {' '.join(mount)} -e NXF_DISABLE_CHECK_LATEST=true {' '.join(proxy)}",
6704        )
6705        # print(docker_cmd)
6706        # exit()
6707        # Docker debug
6708        # if splice_config.get("rm_container"):
6709        #     rm_container = "--rm"
6710        # else:
6711        #     rm_container = ""
6712        # docker_cmd = f"docker run {rm_container} --entrypoint '/bin/bash' --name {random_uuid} {' '.join(mount)} {':'.join(splice_config.get('image'))} {cmd}"
6713        log.debug(docker_cmd)
6714        res = subprocess.run(docker_cmd, shell=True, capture_output=True, text=True)
6715        log.debug(res.stdout)
6716        if res.stderr:
6717            log.error(res.stderr)
6718        res.check_returncode()
6719        # Update variants
6720        log.info("Annotation - Updating...")
6721        # Test find output vcf
6722        log.debug(
6723            f"TMP splice output: {os.path.basename(tmp_vcf_name).replace('.vcf', '')}.spip.spliceai.sorted.vcf.gz"
6724        )
6725        output_vcf = []
6726        # Wrong folder to look in
6727        for files in os.listdir(os.path.dirname(tmp_vcf_name)):
6728            if (
6729                files
6730                == f"{os.path.basename(tmp_vcf_name).replace('.vcf', '')}.spip.spliceai.sorted.vcf.gz"
6731            ):
6732                output_vcf.append(os.path.join(os.path.dirname(tmp_vcf_name), files))
6733        # log.debug(os.listdir(options.get("output_folder")))
6734        log.debug(f"Splice annotated vcf: {output_vcf[0]}")
6735        if not output_vcf:
6736            log.debug(
6737                f"Splice output was not generated {os.path.basename(tmp_vcf_name)}*.spip.spliceai.sorted.vcf.gz"
6738            )
6739        else:
6740            # Get new header from annotated vcf
6741            log.debug(f"Initial header: {len(header.infos)} fields")
6742            # Create new header with splice infos
6743            new_vcf = Variants(input=output_vcf[0])
6744            new_vcf_header = new_vcf.get_header().infos
6745            for keys, infos in new_vcf_header.items():
6746                if keys not in header.infos.keys():
6747                    header.infos[keys] = infos
6748            log.debug(f"New header: {len(header.infos)} fields")
6749            log.debug(f"Splice tmp output: {output_vcf[0]}")
6750            self.update_from_vcf(output_vcf[0])
6751
6752        # Remove file
6753        remove_if_exists(output_vcf)

This function annotate with snpEff

Parameters

threads: The number of threads to use

Returns

the value of the variable "return_value".

def get_config_default(self, name: str) -> dict: View Source

6759    def get_config_default(self, name: str) -> dict:
6760        """
6761        The function `get_config_default` returns a dictionary containing default configurations for
6762        various calculations and prioritizations.
6763
6764        :param name: The `get_config_default` function returns a dictionary containing default
6765        configurations for different calculations and prioritizations. The `name` parameter is used to
6766        specify which specific configuration to retrieve from the dictionary
6767        :type name: str
6768        :return: The function `get_config_default` returns a dictionary containing default configuration
6769        settings for different calculations and prioritizations. The specific configuration settings are
6770        retrieved based on the input `name` parameter provided to the function. If the `name` parameter
6771        matches a key in the `config_default` dictionary, the corresponding configuration settings are
6772        returned. If there is no match, an empty dictionary is returned.
6773        """
6774
6775        config_default = {
6776            "calculations": {
6777                "variant_chr_pos_alt_ref": {
6778                    "type": "sql",
6779                    "name": "variant_chr_pos_alt_ref",
6780                    "description": "Create a variant ID with chromosome, position, alt and ref",
6781                    "available": False,
6782                    "output_column_name": "variant_chr_pos_alt_ref",
6783                    "output_column_type": "String",
6784                    "output_column_description": "variant ID with chromosome, position, alt and ref",
6785                    "operation_query": """ concat("#CHROM", '_', "POS", '_', "REF", '_', "ALT") """,
6786                    "operation_info": True,
6787                },
6788                "VARTYPE": {
6789                    "type": "sql",
6790                    "name": "VARTYPE",
6791                    "description": "Variant type (e.g. SNV, INDEL, MNV, BND...)",
6792                    "available": True,
6793                    "table": "variants",
6794                    "output_column_name": "VARTYPE",
6795                    "output_column_type": "String",
6796                    "output_column_description": "Variant type: SNV if X>Y, MOSAIC if X>Y,Z or X,Y>Z, INDEL if XY>Z or X>YZ",
6797                    "operation_query": """
6798                            CASE
6799                                WHEN "SVTYPE" NOT NULL THEN "SVTYPE"
6800                                WHEN LENGTH(REF) = 1 AND LENGTH(ALT) = 1 THEN 'SNV'
6801                                WHEN REF LIKE '%,%' OR ALT LIKE '%,%' THEN 'MOSAIC'
6802                                WHEN LENGTH(REF) == LENGTH(ALT) AND LENGTH(REF) > 1 THEN 'MNV'
6803                                WHEN LENGTH(REF) <> LENGTH(ALT) THEN 'INDEL'
6804                                ELSE 'UNDEFINED'
6805                            END
6806                            """,
6807                    "info_fields": ["SVTYPE"],
6808                    "operation_info": True,
6809                },
6810                "snpeff_hgvs": {
6811                    "type": "python",
6812                    "name": "snpeff_hgvs",
6813                    "description": "HGVS nomenclatures from snpEff annotation",
6814                    "available": True,
6815                    "function_name": "calculation_extract_snpeff_hgvs",
6816                    "function_params": ["snpeff_hgvs", "ANN"],
6817                },
6818                "snpeff_ann_explode": {
6819                    "type": "python",
6820                    "name": "snpeff_ann_explode",
6821                    "description": "Explode snpEff annotations with uniquify values",
6822                    "available": True,
6823                    "function_name": "calculation_snpeff_ann_explode",
6824                    "function_params": [False, "fields", "snpeff_", "ANN"],
6825                },
6826                "snpeff_ann_explode_uniquify": {
6827                    "type": "python",
6828                    "name": "snpeff_ann_explode_uniquify",
6829                    "description": "Explode snpEff annotations",
6830                    "available": True,
6831                    "function_name": "calculation_snpeff_ann_explode",
6832                    "function_params": [True, "fields", "snpeff_uniquify_", "ANN"],
6833                },
6834                "snpeff_ann_explode_json": {
6835                    "type": "python",
6836                    "name": "snpeff_ann_explode_json",
6837                    "description": "Explode snpEff annotations in JSON format",
6838                    "available": True,
6839                    "function_name": "calculation_snpeff_ann_explode",
6840                    "function_params": [False, "JSON", "snpeff_json", "ANN"],
6841                },
6842                "NOMEN": {
6843                    "type": "python",
6844                    "name": "NOMEN",
6845                    "description": "NOMEN information (e.g. NOMEN, CNOMEN, PNOMEN...) from HGVS nomenclature field (see parameters help)",
6846                    "available": True,
6847                    "function_name": "calculation_extract_nomen",
6848                    "function_params": [],
6849                },
6850                "RENAME_INFO_FIELDS": {
6851                    "type": "python",
6852                    "name": "RENAME_INFO_FIELDS",
6853                    "description": "Rename or remove INFO/tags",
6854                    "available": True,
6855                    "function_name": "calculation_rename_info_fields",
6856                    "function_params": [],
6857                },
6858                "FINDBYPIPELINE": {
6859                    "type": "python",
6860                    "name": "FINDBYPIPELINE",
6861                    "description": "Number of pipeline that identify the variant (for multi pipeline VCF)",
6862                    "available": True,
6863                    "function_name": "calculation_find_by_pipeline",
6864                    "function_params": ["findbypipeline"],
6865                },
6866                "FINDBYSAMPLE": {
6867                    "type": "python",
6868                    "name": "FINDBYSAMPLE",
6869                    "description": "Number of sample that have a genotype for the variant (for multi sample VCF)",
6870                    "available": True,
6871                    "function_name": "calculation_find_by_pipeline",
6872                    "function_params": ["findbysample"],
6873                },
6874                "GENOTYPECONCORDANCE": {
6875                    "type": "python",
6876                    "name": "GENOTYPECONCORDANCE",
6877                    "description": "Concordance of genotype for multi caller VCF",
6878                    "available": True,
6879                    "function_name": "calculation_genotype_concordance",
6880                    "function_params": [],
6881                },
6882                "BARCODE": {
6883                    "type": "python",
6884                    "name": "BARCODE",
6885                    "description": "BARCODE as VaRank tool",
6886                    "available": True,
6887                    "function_name": "calculation_barcode",
6888                    "function_params": [],
6889                },
6890                "BARCODEFAMILY": {
6891                    "type": "python",
6892                    "name": "BARCODEFAMILY",
6893                    "description": "BARCODEFAMILY as VaRank tool",
6894                    "available": True,
6895                    "function_name": "calculation_barcode_family",
6896                    "function_params": ["BCF"],
6897                },
6898                "TRIO": {
6899                    "type": "python",
6900                    "name": "TRIO",
6901                    "description": "Inheritance for a trio family",
6902                    "available": True,
6903                    "function_name": "calculation_trio",
6904                    "function_params": [],
6905                },
6906                "VAF": {
6907                    "type": "python",
6908                    "name": "VAF",
6909                    "description": "Variant Allele Frequency (VAF) harmonization",
6910                    "available": True,
6911                    "function_name": "calculation_vaf_normalization",
6912                    "function_params": [],
6913                },
6914                "VAF_stats": {
6915                    "type": "python",
6916                    "name": "VAF_stats",
6917                    "description": "Variant Allele Frequency (VAF) statistics",
6918                    "available": True,
6919                    "function_name": "calculation_genotype_stats",
6920                    "function_params": ["VAF"],
6921                },
6922                "DP_stats": {
6923                    "type": "python",
6924                    "name": "DP_stats",
6925                    "description": "Depth (DP) statistics",
6926                    "available": True,
6927                    "function_name": "calculation_genotype_stats",
6928                    "function_params": ["DP"],
6929                },
6930                "variant_id": {
6931                    "type": "python",
6932                    "name": "variant_id",
6933                    "description": "Variant ID generated from variant position and type",
6934                    "available": True,
6935                    "function_name": "calculation_variant_id",
6936                    "function_params": [],
6937                },
6938                "transcripts_json": {
6939                    "type": "python",
6940                    "name": "transcripts_json",
6941                    "description": "Add transcripts annotations in JSON format (field 'transcripts_json')",
6942                    "available": True,
6943                    "function_name": "calculation_transcripts_annotation",
6944                    "function_params": ["transcripts_json", None],
6945                },
6946                "transcripts_ann": {
6947                    "type": "python",
6948                    "name": "transcripts_ann",
6949                    "description": "Add transcripts annotations in structured format (field 'transcripts_ann')",
6950                    "available": True,
6951                    "function_name": "calculation_transcripts_annotation",
6952                    "function_params": [None, "transcripts_ann"],
6953                },
6954                "transcripts_annotations": {
6955                    "type": "python",
6956                    "name": "transcripts_annotations",
6957                    "description": "Add transcripts annotations in JSON and/or structured format (see param JSON file)",
6958                    "available": True,
6959                    "function_name": "calculation_transcripts_annotation",
6960                    "function_params": [None, None],
6961                },
6962                "transcripts_prioritization": {
6963                    "type": "python",
6964                    "name": "transcripts_prioritization",
6965                    "description": "Prioritize transcripts with a prioritization profile (using param.json)",
6966                    "available": True,
6967                    "function_name": "calculation_transcripts_prioritization",
6968                    "function_params": [],
6969                },
6970                "transcripts_export": {
6971                    "type": "python",
6972                    "name": "transcripts_export",
6973                    "description": "Export transcripts table/view as a file (using param.json)",
6974                    "available": True,
6975                    "function_name": "calculation_transcripts_export",
6976                    "function_params": [],
6977                },
6978            },
6979            "prioritizations": {
6980                "default": {
6981                    "ANN2": [
6982                        {
6983                            "type": "contains",
6984                            "value": "HIGH",
6985                            "score": 5,
6986                            "flag": "PASS",
6987                            "comment": [
6988                                "The variant is assumed to have high (disruptive) impact in the protein, probably causing protein truncation, loss of function or triggering nonsense mediated decay"
6989                            ],
6990                        },
6991                        {
6992                            "type": "contains",
6993                            "value": "MODERATE",
6994                            "score": 3,
6995                            "flag": "PASS",
6996                            "comment": [
6997                                "A non-disruptive variant that might change protein effectiveness"
6998                            ],
6999                        },
7000                        {
7001                            "type": "contains",
7002                            "value": "LOW",
7003                            "score": 0,
7004                            "flag": "FILTERED",
7005                            "comment": [
7006                                "Assumed to be mostly harmless or unlikely to change protein behavior"
7007                            ],
7008                        },
7009                        {
7010                            "type": "contains",
7011                            "value": "MODIFIER",
7012                            "score": 0,
7013                            "flag": "FILTERED",
7014                            "comment": [
7015                                "Usually non-coding variants or variants affecting non-coding genes, where predictions are difficult or there is no evidence of impact"
7016                            ],
7017                        },
7018                    ],
7019                }
7020            },
7021        }
7022
7023        return config_default.get(name, None)

The function get_config_default returns a dictionary containing default configurations for various calculations and prioritizations.

Parameters

name: The get_config_default function returns a dictionary containing default configurations for different calculations and prioritizations. The name parameter is used to specify which specific configuration to retrieve from the dictionary

Returns

The function get_config_default returns a dictionary containing default configuration settings for different calculations and prioritizations. The specific configuration settings are retrieved based on the input name parameter provided to the function. If the name parameter matches a key in the config_default dictionary, the corresponding configuration settings are returned. If there is no match, an empty dictionary is returned.

def get_config_json(self, name: str, config_dict: dict = {}, config_file: str = None) -> dict: View Source

7025    def get_config_json(
7026        self, name: str, config_dict: dict = {}, config_file: str = None
7027    ) -> dict:
7028        """
7029        The function `get_config_json` retrieves a configuration JSON object with prioritizations from
7030        default values, a dictionary, and a file.
7031
7032        :param name: The `name` parameter in the `get_config_json` function is a string that represents
7033        the name of the configuration. It is used to identify and retrieve the configuration settings
7034        for a specific component or module
7035        :type name: str
7036        :param config_dict: The `config_dict` parameter in the `get_config_json` function is a
7037        dictionary that allows you to provide additional configuration settings or overrides. When you
7038        call the `get_config_json` function, you can pass a dictionary containing key-value pairs where
7039        the key is the configuration setting you want to override or
7040        :type config_dict: dict
7041        :param config_file: The `config_file` parameter in the `get_config_json` function is used to
7042        specify the path to a configuration file that contains additional settings. If provided, the
7043        function will read the contents of this file and update the configuration dictionary with the
7044        values found in the file, overriding any existing values with the
7045        :type config_file: str
7046        :return: The function `get_config_json` returns a dictionary containing the configuration
7047        settings.
7048        """
7049
7050        # Create with default prioritizations
7051        config_default = self.get_config_default(name=name)
7052        configuration = config_default
7053        # log.debug(f"configuration={configuration}")
7054
7055        # Replace prioritizations from dict
7056        for config in config_dict:
7057            configuration[config] = config_dict[config]
7058
7059        # Replace prioritizations from file
7060        config_file = full_path(config_file)
7061        if config_file:
7062            if os.path.exists(config_file):
7063                with open(config_file) as config_file_content:
7064                    config_file_dict = yaml.safe_load(config_file_content)
7065                for config in config_file_dict:
7066                    configuration[config] = config_file_dict[config]
7067            else:
7068                msg_error = f"Config '{name}' file '{config_file}' does NOT exist"
7069                log.error(msg_error)
7070                raise ValueError(msg_error)
7071
7072        return configuration

The function get_config_json retrieves a configuration JSON object with prioritizations from default values, a dictionary, and a file.

Parameters

name: The name parameter in the get_config_json function is a string that represents the name of the configuration. It is used to identify and retrieve the configuration settings for a specific component or module
config_dict: The config_dict parameter in the get_config_json function is a dictionary that allows you to provide additional configuration settings or overrides. When you call the get_config_json function, you can pass a dictionary containing key-value pairs where the key is the configuration setting you want to override or
config_file: The config_file parameter in the get_config_json function is used to specify the path to a configuration file that contains additional settings. If provided, the function will read the contents of this file and update the configuration dictionary with the values found in the file, overriding any existing values with the

Returns

The function get_config_json returns a dictionary containing the configuration settings.

def prioritization( self, table: str = None, pz_prefix: str = None, pz_param: dict = None) -> bool: View Source

7074    def prioritization(
7075        self, table: str = None, pz_prefix: str = None, pz_param: dict = None
7076    ) -> bool:
7077        """
7078        The `prioritization` function in Python processes VCF files, adds new INFO fields, and
7079        prioritizes variants based on configured profiles and criteria.
7080
7081        :param table: The `table` parameter in the `prioritization` function is used to specify the name
7082        of the table (presumably a VCF file) on which the prioritization operation will be performed. If
7083        a table name is provided, the method will prioritize the variants in that specific table
7084        :type table: str
7085        :param pz_prefix: The `pz_prefix` parameter is used to specify a prefix that will be added to
7086        certain INFO fields in a VCF file during the prioritization process. If this parameter is not
7087        provided, the code will use a default prefix value of "PZ"
7088        :type pz_prefix: str
7089        :param pz_param: The `pz_param` parameter in the `prioritization` method is used to pass
7090        additional parameters specific to the prioritization process. These parameters can include
7091        settings related to prioritization profiles, fields, scoring modes, flags, comments, and other
7092        configurations needed for the prioritization of variants in a V
7093        :type pz_param: dict
7094        :return: A boolean value (True) is being returned from the `prioritization` function.
7095        """
7096
7097        # Config
7098        config = self.get_config()
7099
7100        # Param
7101        param = self.get_param()
7102
7103        # Prioritization param
7104        if pz_param is not None:
7105            prioritization_param = pz_param
7106        else:
7107            prioritization_param = param.get("prioritization", {})
7108
7109        # Configuration profiles
7110        prioritization_config_file = prioritization_param.get(
7111            "prioritization_config", None
7112        )
7113        prioritization_config_file = full_path(prioritization_config_file)
7114        prioritizations_config = self.get_config_json(
7115            name="prioritizations", config_file=prioritization_config_file
7116        )
7117
7118        # Prioritization prefix
7119        pz_prefix_default = "PZ"
7120        if pz_prefix is None:
7121            pz_prefix = prioritization_param.get("pzprefix", pz_prefix_default)
7122
7123        # Prioritization options
7124        profiles = prioritization_param.get("profiles", [])
7125        if isinstance(profiles, str):
7126            profiles = profiles.split(",")
7127        pzfields = prioritization_param.get(
7128            "pzfields", [f"{pz_prefix}Flag", f"{pz_prefix}Score"]
7129        )
7130        if isinstance(pzfields, str):
7131            pzfields = pzfields.split(",")
7132        default_profile = prioritization_param.get("default_profile", None)
7133        pzfields_sep = prioritization_param.get("pzfields_sep", "_")
7134        prioritization_score_mode = prioritization_param.get(
7135            "prioritization_score_mode", "HOWARD"
7136        )
7137
7138        # Quick Prioritizations
7139        prioritizations = param.get("prioritizations", None)
7140        if prioritizations:
7141            log.info("Quick Prioritization:")
7142            for profile in prioritizations.split(","):
7143                if profile not in profiles:
7144                    profiles.append(profile)
7145                    log.info(f"   {profile}")
7146
7147        # If profile "ALL" provided, all profiles in the config profiles
7148        if "ALL" in profiles:
7149            profiles = list(prioritizations_config.keys())
7150
7151        for profile in profiles:
7152            if prioritizations_config.get(profile, None):
7153                log.debug(f"Profile '{profile}' configured")
7154            else:
7155                msg_error = f"Profile '{profile}' NOT configured"
7156                log.error(msg_error)
7157                raise ValueError(msg_error)
7158
7159        if profiles:
7160            log.info(f"Prioritization... ")
7161        else:
7162            log.debug(f"No profile defined")
7163            return False
7164
7165        if not default_profile and len(profiles):
7166            default_profile = profiles[0]
7167
7168        log.debug("Profiles availables: " + str(list(prioritizations_config.keys())))
7169        log.debug("Profiles to check: " + str(list(profiles)))
7170
7171        # Variables
7172        if table is not None:
7173            table_variants = table
7174        else:
7175            table_variants = self.get_table_variants(clause="update")
7176        log.debug(f"Table to prioritize: {table_variants}")
7177
7178        # Added columns
7179        added_columns = []
7180
7181        # Create list of PZfields
7182        # List of PZFields
7183        list_of_pzfields_original = pzfields + [
7184            pzfield + pzfields_sep + profile
7185            for pzfield in pzfields
7186            for profile in profiles
7187        ]
7188        list_of_pzfields = []
7189        log.debug(f"{list_of_pzfields_original}")
7190
7191        # Remove existing PZfields to use if exists
7192        for pzfield in list_of_pzfields_original:
7193            if self.get_header().infos.get(pzfield, None) is None:
7194                list_of_pzfields.append(pzfield)
7195                log.debug(f"VCF Input - Header - PZfield '{pzfield}' not in VCF")
7196            else:
7197                log.debug(f"VCF Input - Header - PZfield '{pzfield}' already in VCF")
7198
7199        if list_of_pzfields:
7200
7201            # Explode Infos prefix
7202            explode_infos_prefix = self.get_explode_infos_prefix()
7203
7204            # PZfields tags description
7205            PZfields_INFOS = {
7206                f"{pz_prefix}Tags": {
7207                    "ID": f"{pz_prefix}Tags",
7208                    "Number": ".",
7209                    "Type": "String",
7210                    "Description": "Variant tags based on annotation criteria",
7211                },
7212                f"{pz_prefix}Score": {
7213                    "ID": f"{pz_prefix}Score",
7214                    "Number": 1,
7215                    "Type": "Integer",
7216                    "Description": "Variant score based on annotation criteria",
7217                },
7218                f"{pz_prefix}Flag": {
7219                    "ID": f"{pz_prefix}Flag",
7220                    "Number": 1,
7221                    "Type": "String",
7222                    "Description": "Variant flag based on annotation criteria",
7223                },
7224                f"{pz_prefix}Comment": {
7225                    "ID": f"{pz_prefix}Comment",
7226                    "Number": ".",
7227                    "Type": "String",
7228                    "Description": "Variant comment based on annotation criteria",
7229                },
7230                f"{pz_prefix}Infos": {
7231                    "ID": f"{pz_prefix}Infos",
7232                    "Number": ".",
7233                    "Type": "String",
7234                    "Description": "Variant infos based on annotation criteria",
7235                },
7236                f"{pz_prefix}Class": {
7237                    "ID": f"{pz_prefix}Class",
7238                    "Number": ".",
7239                    "Type": "String",
7240                    "Description": "Variant class based on annotation criteria",
7241                },
7242            }
7243
7244            # Create INFO fields if not exist
7245            for field in PZfields_INFOS:
7246                field_ID = PZfields_INFOS[field]["ID"]
7247                field_description = PZfields_INFOS[field]["Description"]
7248                if field_ID not in self.get_header().infos and field_ID in pzfields:
7249                    field_description = (
7250                        PZfields_INFOS[field]["Description"]
7251                        + f", profile {default_profile}"
7252                    )
7253                    self.get_header().infos[field_ID] = vcf.parser._Info(
7254                        field_ID,
7255                        PZfields_INFOS[field]["Number"],
7256                        PZfields_INFOS[field]["Type"],
7257                        field_description,
7258                        "unknown",
7259                        "unknown",
7260                        code_type_map[PZfields_INFOS[field]["Type"]],
7261                    )
7262
7263            # Create INFO fields if not exist for each profile
7264            for profile in prioritizations_config:
7265                if profile in profiles or profiles == []:
7266                    for field in PZfields_INFOS:
7267                        field_ID = PZfields_INFOS[field]["ID"] + pzfields_sep + profile
7268                        field_description = (
7269                            PZfields_INFOS[field]["Description"]
7270                            + f", profile {profile}"
7271                        )
7272                        if (
7273                            field_ID not in self.get_header().infos
7274                            and field in pzfields
7275                        ):
7276                            self.get_header().infos[field_ID] = vcf.parser._Info(
7277                                field_ID,
7278                                PZfields_INFOS[field]["Number"],
7279                                PZfields_INFOS[field]["Type"],
7280                                field_description,
7281                                "unknown",
7282                                "unknown",
7283                                code_type_map[PZfields_INFOS[field]["Type"]],
7284                            )
7285
7286            # Header
7287            for pzfield in list_of_pzfields:
7288                if re.match(f"{pz_prefix}Score.*", pzfield):
7289                    added_column = self.add_column(
7290                        table_name=table_variants,
7291                        column_name=pzfield,
7292                        column_type="INTEGER",
7293                        default_value="0",
7294                    )
7295                elif re.match(f"{pz_prefix}Flag.*", pzfield):
7296                    added_column = self.add_column(
7297                        table_name=table_variants,
7298                        column_name=pzfield,
7299                        column_type="BOOLEAN",
7300                        default_value="1",
7301                    )
7302                elif re.match(f"{pz_prefix}Class.*", pzfield):
7303                    added_column = self.add_column(
7304                        table_name=table_variants,
7305                        column_name=pzfield,
7306                        column_type="VARCHAR[]",
7307                        default_value="null",
7308                    )
7309                else:
7310                    added_column = self.add_column(
7311                        table_name=table_variants,
7312                        column_name=pzfield,
7313                        column_type="STRING",
7314                        default_value="''",
7315                    )
7316                added_columns.append(added_column)
7317
7318            # Profiles
7319            if profiles:
7320
7321                # foreach profile in configuration file
7322                for profile in prioritizations_config:
7323
7324                    # If profile is asked in param, or ALL are asked (empty profile [])
7325                    if profile in profiles or profiles == []:
7326                        log.info(f"Profile '{profile}'")
7327
7328                        sql_set_info_option = ""
7329
7330                        sql_set_info = []
7331
7332                        # PZ fields set
7333
7334                        # PZScore
7335                        if (
7336                            f"{pz_prefix}Score{pzfields_sep}{profile}"
7337                            in list_of_pzfields
7338                        ):
7339                            sql_set_info.append(
7340                                f"""
7341                                    concat(
7342                                        '{pz_prefix}Score{pzfields_sep}{profile}=',
7343                                        {pz_prefix}Score{pzfields_sep}{profile}
7344                                    ) 
7345                                """
7346                            )
7347                            if (
7348                                profile == default_profile
7349                                and f"{pz_prefix}Score" in list_of_pzfields
7350                            ):
7351                                sql_set_info.append(
7352                                    f"""
7353                                        concat(
7354                                            '{pz_prefix}Score=',
7355                                            {pz_prefix}Score{pzfields_sep}{profile}
7356                                        )
7357                                    """
7358                                )
7359
7360                        # PZFlag
7361                        if (
7362                            f"{pz_prefix}Flag{pzfields_sep}{profile}"
7363                            in list_of_pzfields
7364                        ):
7365                            sql_set_info.append(
7366                                f"""
7367                                    concat(
7368                                        '{pz_prefix}Flag{pzfields_sep}{profile}=',
7369                                        CASE 
7370                                            WHEN {pz_prefix}Flag{pzfields_sep}{profile}==1
7371                                            THEN 'PASS'
7372                                            WHEN {pz_prefix}Flag{pzfields_sep}{profile}==0
7373                                            THEN 'FILTERED'
7374                                        END
7375                                    ) 
7376                                """
7377                            )
7378                            if (
7379                                profile == default_profile
7380                                and f"{pz_prefix}Flag" in list_of_pzfields
7381                            ):
7382                                sql_set_info.append(
7383                                    f"""
7384                                        concat(
7385                                            '{pz_prefix}Flag=',
7386                                            CASE 
7387                                                WHEN {pz_prefix}Flag{pzfields_sep}{profile}==1
7388                                                THEN 'PASS'
7389                                                WHEN {pz_prefix}Flag{pzfields_sep}{profile}==0
7390                                                THEN 'FILTERED'
7391                                            END
7392                                        )
7393                                    """
7394                                )
7395
7396                        # PZClass
7397                        if (
7398                            f"{pz_prefix}Class{pzfields_sep}{profile}"
7399                            in list_of_pzfields
7400                        ):
7401                            sql_set_info.append(
7402                                f"""
7403                                    concat(
7404                                        '{pz_prefix}Class{pzfields_sep}{profile}=',
7405                                        CASE
7406                                            WHEN len({pz_prefix}Class{pzfields_sep}{profile}) > 0
7407                                            THEN list_aggregate(list_distinct({pz_prefix}Class{pzfields_sep}{profile}), 'string_agg', ',')
7408                                            ELSE '.'
7409                                        END 
7410                                    )
7411                                    
7412                                """
7413                            )
7414                            if (
7415                                profile == default_profile
7416                                and f"{pz_prefix}Class" in list_of_pzfields
7417                            ):
7418                                sql_set_info.append(
7419                                    f"""
7420                                        concat(
7421                                            '{pz_prefix}Class=',
7422                                            CASE
7423                                                WHEN len({pz_prefix}Class{pzfields_sep}{profile}) > 0
7424                                                THEN list_aggregate(list_distinct({pz_prefix}Class{pzfields_sep}{profile}), 'string_agg', ',')
7425                                                ELSE '.'
7426                                            END 
7427                                        )
7428                                    """
7429                                )
7430
7431                        # PZComment
7432                        if (
7433                            f"{pz_prefix}Comment{pzfields_sep}{profile}"
7434                            in list_of_pzfields
7435                        ):
7436                            sql_set_info.append(
7437                                f"""
7438                                    CASE
7439                                        WHEN {pz_prefix}Comment{pzfields_sep}{profile} NOT IN ('')
7440                                        THEN concat('{pz_prefix}Comment{pzfields_sep}{profile}=', {pz_prefix}Comment{pzfields_sep}{profile})
7441                                        ELSE ''
7442                                    END
7443                                """
7444                            )
7445                            if (
7446                                profile == default_profile
7447                                and f"{pz_prefix}Comment" in list_of_pzfields
7448                            ):
7449                                sql_set_info.append(
7450                                    f"""
7451                                        CASE
7452                                            WHEN {pz_prefix}Comment{pzfields_sep}{profile} NOT IN ('')
7453                                            THEN concat('{pz_prefix}Comment=', {pz_prefix}Comment{pzfields_sep}{profile})
7454                                            ELSE ''
7455                                        END
7456                                    """
7457                                )
7458
7459                        # PZInfos
7460                        if (
7461                            f"{pz_prefix}Infos{pzfields_sep}{profile}"
7462                            in list_of_pzfields
7463                        ):
7464                            sql_set_info.append(
7465                                f"""
7466                                    CASE
7467                                        WHEN {pz_prefix}Infos{pzfields_sep}{profile} NOT IN ('')
7468                                        THEN concat('{pz_prefix}Infos{pzfields_sep}{profile}=', {pz_prefix}Infos{pzfields_sep}{profile})
7469                                        ELSE ''
7470                                    END
7471                                """
7472                            )
7473                            if (
7474                                profile == default_profile
7475                                and f"{pz_prefix}Infos" in list_of_pzfields
7476                            ):
7477                                sql_set_info.append(
7478                                    f"""
7479                                        CASE
7480                                            WHEN {pz_prefix}Infos{pzfields_sep}{profile} NOT IN ('')
7481                                            THEN concat('{pz_prefix}Infos=', {pz_prefix}Infos{pzfields_sep}{profile})
7482                                            ELSE ''
7483                                        END
7484                                    """
7485                                )
7486
7487                        # Merge PZfields
7488                        sql_set_info_option = ""
7489                        sql_set_sep = ""
7490                        for sql_set in sql_set_info:
7491                            if sql_set_sep:
7492                                sql_set_info_option += f"""
7493                                    , concat('{sql_set_sep}', {sql_set})
7494                                """
7495                            else:
7496                                sql_set_info_option += f"""
7497                                    , {sql_set}
7498                                """
7499                            sql_set_sep = ";"
7500
7501                        sql_queries = []
7502                        for annotation in prioritizations_config[profile]:
7503
7504                            # skip special sections
7505                            if annotation.startswith("_"):
7506                                continue
7507
7508                            # For each criterions
7509                            for criterion in prioritizations_config[profile][
7510                                annotation
7511                            ]:
7512
7513                                # Criterion mode
7514                                criterion_mode = None
7515                                if np.any(
7516                                    np.isin(list(criterion.keys()), ["type", "value"])
7517                                ):
7518                                    criterion_mode = "operation"
7519                                elif np.any(
7520                                    np.isin(list(criterion.keys()), ["sql", "fields"])
7521                                ):
7522                                    criterion_mode = "sql"
7523                                log.debug(f"Criterion Mode: {criterion_mode}")
7524
7525                                # Criterion parameters
7526                                criterion_type = criterion.get("type", None)
7527                                criterion_value = criterion.get("value", None)
7528                                criterion_sql = criterion.get("sql", None)
7529                                criterion_fields = criterion.get("fields", None)
7530                                criterion_score = criterion.get("score", 0)
7531                                criterion_flag = criterion.get("flag", "PASS")
7532                                criterion_class = criterion.get("class", None)
7533                                criterion_flag_bool = criterion_flag == "PASS"
7534                                criterion_comment = (
7535                                    ", ".join(criterion.get("comment", []))
7536                                    .replace("'", "''")
7537                                    .replace(";", ",")
7538                                    .replace("\t", " ")
7539                                )
7540                                criterion_infos = (
7541                                    str(criterion)
7542                                    .replace("'", "''")
7543                                    .replace(";", ",")
7544                                    .replace("\t", " ")
7545                                )
7546
7547                                # SQL
7548                                if criterion_sql is not None and isinstance(
7549                                    criterion_sql, list
7550                                ):
7551                                    criterion_sql = " ".join(criterion_sql)
7552
7553                                # Fields and explode
7554                                if criterion_fields is None:
7555                                    criterion_fields = [annotation]
7556                                if not isinstance(criterion_fields, list):
7557                                    criterion_fields = str(criterion_fields).split(",")
7558
7559                                # Class
7560                                if criterion_class is not None and not isinstance(
7561                                    criterion_class, list
7562                                ):
7563                                    criterion_class = str(criterion_class).split(",")
7564
7565                                for annotation_field in criterion_fields:
7566
7567                                    # Explode specific annotation
7568                                    log.debug(
7569                                        f"Explode annotation '{annotation_field}'"
7570                                    )
7571                                    added_columns += self.explode_infos(
7572                                        prefix=explode_infos_prefix,
7573                                        fields=[annotation_field],
7574                                        table=table_variants,
7575                                    )
7576                                    extra_infos = self.get_extra_infos(
7577                                        table=table_variants
7578                                    )
7579
7580                                    # Check if annotation field is present
7581                                    if (
7582                                        f"{explode_infos_prefix}{annotation_field}"
7583                                        not in extra_infos
7584                                    ):
7585                                        msq_err = f"Annotation '{annotation_field}' not in data"
7586                                        log.error(msq_err)
7587                                        raise ValueError(msq_err)
7588                                    else:
7589                                        log.debug(
7590                                            f"Annotation '{annotation_field}' in data"
7591                                        )
7592
7593                                sql_set = []
7594                                sql_set_info = []
7595
7596                                # PZ fields set
7597
7598                                # PZScore
7599                                if (
7600                                    f"{pz_prefix}Score{pzfields_sep}{profile}"
7601                                    in list_of_pzfields
7602                                ):
7603                                    # VaRank prioritization score mode
7604                                    if prioritization_score_mode.upper().strip() in [
7605                                        "VARANK",
7606                                        "MAX",
7607                                        "MAXIMUM",
7608                                        "TOP",
7609                                    ]:
7610                                        sql_set.append(
7611                                            f"{pz_prefix}Score{pzfields_sep}{profile} = CASE WHEN {criterion_score}>{pz_prefix}Score{pzfields_sep}{profile} THEN {criterion_score} ELSE {pz_prefix}Score{pzfields_sep}{profile} END "
7612                                        )
7613                                    # default HOWARD prioritization score mode
7614                                    else:
7615                                        sql_set.append(
7616                                            f"{pz_prefix}Score{pzfields_sep}{profile} = {pz_prefix}Score{pzfields_sep}{profile} + {criterion_score}"
7617                                        )
7618
7619                                # PZFlag
7620                                if (
7621                                    f"{pz_prefix}Flag{pzfields_sep}{profile}"
7622                                    in list_of_pzfields
7623                                ):
7624                                    sql_set.append(
7625                                        f"{pz_prefix}Flag{pzfields_sep}{profile} = {pz_prefix}Flag{pzfields_sep}{profile} AND {criterion_flag_bool}"
7626                                    )
7627
7628                                # PZClass
7629                                if (
7630                                    f"{pz_prefix}Class{pzfields_sep}{profile}"
7631                                    in list_of_pzfields
7632                                    and criterion_class is not None
7633                                ):
7634                                    sql_set.append(
7635                                        f" {pz_prefix}Class{pzfields_sep}{profile} = list_concat(list_distinct({pz_prefix}Class{pzfields_sep}{profile}), {criterion_class}) "
7636                                    )
7637
7638                                # PZComment
7639                                if (
7640                                    f"{pz_prefix}Comment{pzfields_sep}{profile}"
7641                                    in list_of_pzfields
7642                                ):
7643                                    sql_set.append(
7644                                        f"""
7645                                            {pz_prefix}Comment{pzfields_sep}{profile} = 
7646                                                concat(
7647                                                    {pz_prefix}Comment{pzfields_sep}{profile},
7648                                                    CASE 
7649                                                        WHEN {pz_prefix}Comment{pzfields_sep}{profile}!=''
7650                                                        THEN ', '
7651                                                        ELSE ''
7652                                                    END,
7653                                                    '{criterion_comment}'
7654                                                )
7655                                        """
7656                                    )
7657
7658                                # PZInfos
7659                                if (
7660                                    f"{pz_prefix}Infos{pzfields_sep}{profile}"
7661                                    in list_of_pzfields
7662                                ):
7663                                    sql_set.append(
7664                                        f"""
7665                                            {pz_prefix}Infos{pzfields_sep}{profile} = 
7666                                                concat(
7667                                                    {pz_prefix}Infos{pzfields_sep}{profile},
7668                                                    '{criterion_infos}'
7669                                                )
7670                                        """
7671                                    )
7672                                sql_set_option = ",".join(sql_set)
7673
7674                                # Criterion and comparison
7675                                if sql_set_option:
7676
7677                                    if criterion_mode in ["operation"]:
7678
7679                                        try:
7680                                            float(criterion_value)
7681                                            sql_update = f"""
7682                                                UPDATE {table_variants}
7683                                                SET {sql_set_option}
7684                                                WHERE CAST("{explode_infos_prefix}{annotation}" AS VARCHAR) NOT IN ('','.')
7685                                                AND CAST("{explode_infos_prefix}{annotation}" AS FLOAT){comparison_map[criterion_type]}{criterion_value}
7686                                            """
7687                                        except:
7688                                            contains_option = ""
7689                                            if criterion_type == "contains":
7690                                                contains_option = ".*"
7691                                            sql_update = f"""
7692                                                UPDATE {table_variants}
7693                                                SET {sql_set_option}
7694                                                WHERE "{explode_infos_prefix}{annotation}" SIMILAR TO '{contains_option}{criterion_value}{contains_option}'
7695                                            """
7696                                        sql_queries.append(sql_update)
7697
7698                                    elif criterion_mode in ["sql"]:
7699
7700                                        sql_update = f"""
7701                                            UPDATE {table_variants}
7702                                            SET {sql_set_option}
7703                                            WHERE {criterion_sql}
7704                                        """
7705                                        sql_queries.append(sql_update)
7706
7707                                    else:
7708                                        msg_err = f"Prioritization criterion mode failed (either 'operation' or 'sql')"
7709                                        log.error(msg_err)
7710                                        raise ValueError(msg_err)
7711
7712                                else:
7713                                    log.warning(
7714                                        f"NO SQL SET option for '{annotation}' - '{criterion}'"
7715                                    )
7716
7717                        # PZTags
7718                        if (
7719                            f"{pz_prefix}Tags{pzfields_sep}{profile}"
7720                            in list_of_pzfields
7721                        ):
7722
7723                            # Create PZFalgs value
7724                            pztags_value = ""
7725                            pztags_sep_default = ","
7726                            pztags_sep = ""
7727                            for pzfield in pzfields:
7728                                if pzfield not in [f"{pz_prefix}Tags"]:
7729                                    if (
7730                                        f"{pzfield}{pzfields_sep}{profile}"
7731                                        in list_of_pzfields
7732                                    ):
7733                                        if pzfield in [f"{pz_prefix}Flag"]:
7734                                            pztags_value += f"""{pztags_sep}{pzfield}#', 
7735                                                CASE WHEN {pz_prefix}Flag{pzfields_sep}{profile}
7736                                                    THEN 'PASS'
7737                                                    ELSE 'FILTERED'
7738                                                END, '"""
7739                                        elif pzfield in [f"{pz_prefix}Class"]:
7740                                            pztags_value += f"""{pztags_sep}{pzfield}#', 
7741                                                CASE WHEN len({pz_prefix}Class{pzfields_sep}{profile}) > 0
7742                                                    THEN list_aggregate(list_distinct({pz_prefix}Class{pzfields_sep}{profile}), 'string_agg', ',')
7743                                                    ELSE '.'
7744                                                END, '"""
7745                                        else:
7746                                            pztags_value += f"{pztags_sep}{pzfield}#', {pzfield}{pzfields_sep}{profile}, '"
7747                                        pztags_sep = pztags_sep_default
7748
7749                            # Add Query update for PZFlags
7750                            sql_update_pztags = f"""
7751                                UPDATE {table_variants}
7752                                SET INFO = concat(
7753                                        INFO,
7754                                        CASE WHEN INFO NOT in ('','.')
7755                                                THEN ';'
7756                                                ELSE ''
7757                                        END,
7758                                        '{pz_prefix}Tags{pzfields_sep}{profile}={pztags_value}'
7759                                    )
7760                                """
7761                            sql_queries.append(sql_update_pztags)
7762
7763                            # Add Query update for PZFlags for default
7764                            if profile == default_profile:
7765                                sql_update_pztags_default = f"""
7766                                UPDATE {table_variants}
7767                                SET INFO = concat(
7768                                        INFO,
7769                                        ';',
7770                                        '{pz_prefix}Tags={pztags_value}'
7771                                    )
7772                                """
7773                                sql_queries.append(sql_update_pztags_default)
7774
7775                        log.info(f"""Profile '{profile}' - Prioritization... """)
7776
7777                        if sql_queries:
7778
7779                            for sql_query in sql_queries:
7780                                log.debug(
7781                                    f"""Profile '{profile}' - Prioritization query: {sql_query}... """
7782                                )
7783                                self.conn.execute(sql_query)
7784
7785                        log.info(f"""Profile '{profile}' - Update... """)
7786                        sql_query_update = f"""
7787                            UPDATE {table_variants}
7788                            SET INFO =  
7789                                concat(
7790                                    CASE
7791                                        WHEN INFO NOT IN ('','.')
7792                                        THEN concat(INFO, ';')
7793                                        ELSE ''
7794                                    END
7795                                    {sql_set_info_option}
7796                                )
7797                        """
7798                        self.conn.execute(sql_query_update)
7799
7800        else:
7801
7802            log.warning(f"No profiles in parameters")
7803
7804        # Remove added columns
7805        for added_column in added_columns:
7806            self.drop_column(column=added_column)
7807
7808        # Explode INFOS fields into table fields
7809        if self.get_explode_infos():
7810            self.explode_infos(
7811                prefix=self.get_explode_infos_prefix(),
7812                fields=self.get_explode_infos_fields(),
7813                force=True,
7814            )
7815
7816        return True

The prioritization function in Python processes VCF files, adds new INFO fields, and prioritizes variants based on configured profiles and criteria.

Parameters

table: The table parameter in the prioritization function is used to specify the name of the table (presumably a VCF file) on which the prioritization operation will be performed. If a table name is provided, the method will prioritize the variants in that specific table
pz_prefix: The pz_prefix parameter is used to specify a prefix that will be added to certain INFO fields in a VCF file during the prioritization process. If this parameter is not provided, the code will use a default prefix value of "PZ"
pz_param: The pz_param parameter in the prioritization method is used to pass additional parameters specific to the prioritization process. These parameters can include settings related to prioritization profiles, fields, scoring modes, flags, comments, and other configurations needed for the prioritization of variants in a V

Returns

A boolean value (True) is being returned from the prioritization function.

def annotation_hgvs(self, threads: int = None) -> None: View Source

7822    def annotation_hgvs(self, threads: int = None) -> None:
7823        """
7824        The `annotation_hgvs` function performs HGVS annotation on a set of variants using genomic
7825        coordinates and alleles.
7826
7827        :param threads: The `threads` parameter is an optional integer that specifies the number of
7828        threads to use for parallel processing. If no value is provided, it will default to the number
7829        of threads obtained from the `get_threads()` method
7830        :type threads: int
7831        """
7832
7833        # Function for each partition of the Dask Dataframe
7834        def partition_function(partition):
7835            """
7836            The function `partition_function` applies the `annotation_hgvs_partition` function to
7837            each row of a DataFrame called `partition`.
7838
7839            :param partition: The parameter "partition" is a pandas DataFrame that contains the data
7840            to be processed
7841            :return: the result of applying the "annotation_hgvs_partition" function to each row of
7842            the "partition" dataframe along the axis 1.
7843            """
7844            return partition.apply(annotation_hgvs_partition, axis=1)
7845
7846        def annotation_hgvs_partition(row) -> str:
7847            """
7848            The function `annotation_hgvs_partition` takes in a row of data and returns a string
7849            containing a list of HGVS names associated with the given genomic coordinates and alleles.
7850
7851            :param row: A dictionary-like object that contains the values for the following keys:
7852            :return: a string that contains the HGVS names associated with the given row of data.
7853            """
7854
7855            chr = row["CHROM"]
7856            pos = row["POS"]
7857            ref = row["REF"]
7858            alt = row["ALT"]
7859
7860            # Find list of associated transcripts
7861            transcripts_list = list(
7862                polars_conn.execute(
7863                    f"""
7864                SELECT transcript
7865                FROM refseq_df
7866                WHERE CHROM='{chr}'
7867                AND POS={pos}
7868            """
7869                )["transcript"]
7870            )
7871
7872            # Full HGVS annotation in list
7873            hgvs_full_list = []
7874
7875            for transcript_name in transcripts_list:
7876
7877                # Transcript
7878                transcript = get_transcript(
7879                    transcripts=transcripts, transcript_name=transcript_name
7880                )
7881                # Exon
7882                if use_exon:
7883                    exon = transcript.find_exon_number(pos)
7884                else:
7885                    exon = None
7886                # Protein
7887                transcript_protein = None
7888                if use_protein or add_protein or full_format:
7889                    transcripts_protein = list(
7890                        polars_conn.execute(
7891                            f"""
7892                        SELECT protein
7893                        FROM refseqlink_df
7894                        WHERE transcript='{transcript_name}'
7895                        LIMIT 1
7896                    """
7897                        )["protein"]
7898                    )
7899                    if len(transcripts_protein):
7900                        transcript_protein = transcripts_protein[0]
7901
7902                # HGVS name
7903                hgvs_name = format_hgvs_name(
7904                    chr,
7905                    pos,
7906                    ref,
7907                    alt,
7908                    genome=genome,
7909                    transcript=transcript,
7910                    transcript_protein=transcript_protein,
7911                    exon=exon,
7912                    use_gene=use_gene,
7913                    use_protein=use_protein,
7914                    full_format=full_format,
7915                    use_version=use_version,
7916                    codon_type=codon_type,
7917                )
7918                hgvs_full_list.append(hgvs_name)
7919                if add_protein and not use_protein and not full_format:
7920                    hgvs_name = format_hgvs_name(
7921                        chr,
7922                        pos,
7923                        ref,
7924                        alt,
7925                        genome=genome,
7926                        transcript=transcript,
7927                        transcript_protein=transcript_protein,
7928                        exon=exon,
7929                        use_gene=use_gene,
7930                        use_protein=True,
7931                        full_format=False,
7932                        use_version=use_version,
7933                        codon_type=codon_type,
7934                    )
7935                    hgvs_full_list.append(hgvs_name)
7936
7937            # Create liste of HGVS annotations
7938            hgvs_full = ",".join(hgvs_full_list)
7939
7940            return hgvs_full
7941
7942        # Polars connexion
7943        polars_conn = pl.SQLContext(register_globals=True, eager=True)
7944
7945        # Config
7946        config = self.get_config()
7947
7948        # Databases
7949        # Genome
7950        databases_genomes_folders = (
7951            config.get("folders", {})
7952            .get("databases", {})
7953            .get("genomes", DEFAULT_GENOME_FOLDER)
7954        )
7955        databases_genome = (
7956            config.get("folders", {}).get("databases", {}).get("genomes", "")
7957        )
7958        # refseq database folder
7959        databases_refseq_folders = (
7960            config.get("folders", {})
7961            .get("databases", {})
7962            .get("refseq", DEFAULT_REFSEQ_FOLDER)
7963        )
7964        # refseq
7965        databases_refseq = config.get("databases", {}).get("refSeq", None)
7966        # refSeqLink
7967        databases_refseqlink = config.get("databases", {}).get("refSeqLink", None)
7968
7969        # Param
7970        param = self.get_param()
7971
7972        # Quick HGVS
7973        if "hgvs_options" in param and param.get("hgvs_options", ""):
7974            log.info(f"Quick HGVS Annotation:")
7975            if not param.get("hgvs", None):
7976                param["hgvs"] = {}
7977            for option in param.get("hgvs_options", "").split(","):
7978                option_var_val = option.split("=")
7979                option_var = option_var_val[0]
7980                if len(option_var_val) > 1:
7981                    option_val = option_var_val[1]
7982                else:
7983                    option_val = "True"
7984                if option_val.upper() in ["TRUE"]:
7985                    option_val = True
7986                elif option_val.upper() in ["FALSE"]:
7987                    option_val = False
7988                log.info(f"   {option_var}={option_val}")
7989                param["hgvs"][option_var] = option_val
7990
7991        # Check if HGVS annotation enabled
7992        if "hgvs" in param:
7993            log.info(f"HGVS Annotation... ")
7994            for hgvs_option in param.get("hgvs", {}):
7995                log.info(f"{hgvs_option}: {param.get('hgvs',{}).get(hgvs_option)}")
7996        else:
7997            return
7998
7999        # HGVS Param
8000        param_hgvs = param.get("hgvs", {})
8001        use_exon = param_hgvs.get("use_exon", False)
8002        use_gene = param_hgvs.get("use_gene", False)
8003        use_protein = param_hgvs.get("use_protein", False)
8004        add_protein = param_hgvs.get("add_protein", False)
8005        full_format = param_hgvs.get("full_format", False)
8006        use_version = param_hgvs.get("use_version", False)
8007        codon_type = param_hgvs.get("codon_type", "3")
8008
8009        # refSseq refSeqLink
8010        databases_refseq = param_hgvs.get("refseq", databases_refseq)
8011        databases_refseqlink = param_hgvs.get("refseqlink", databases_refseqlink)
8012
8013        # Assembly
8014        assembly = param.get("assembly", config.get("assembly", DEFAULT_ASSEMBLY))
8015
8016        # Genome
8017        genome_file = None
8018        if find_genome(databases_genome):
8019            genome_file = find_genome(databases_genome)
8020        else:
8021            genome_file = find_genome(
8022                genome_path=databases_genomes_folders, assembly=assembly
8023            )
8024        log.debug("Genome: " + str(genome_file))
8025
8026        # refSseq
8027        refseq_file = find_file_prefix(
8028            input_file=databases_refseq,
8029            prefix="ncbiRefSeq",
8030            folder=databases_refseq_folders,
8031            assembly=assembly,
8032        )
8033        log.debug("refSeq: " + str(refseq_file))
8034
8035        # refSeqLink
8036        refseqlink_file = find_file_prefix(
8037            input_file=databases_refseqlink,
8038            prefix="ncbiRefSeqLink",
8039            folder=databases_refseq_folders,
8040            assembly=assembly,
8041        )
8042        log.debug("refSeqLink: " + str(refseqlink_file))
8043
8044        # Threads
8045        if not threads:
8046            threads = self.get_threads()
8047        log.debug("Threads: " + str(threads))
8048
8049        # Variables
8050        table_variants = self.get_table_variants(clause="update")
8051
8052        # Get variants SNV and InDel only
8053        query_variants = f"""
8054            SELECT "#CHROM" AS CHROM, POS, REF, ALT
8055            FROM {table_variants}
8056            WHERE REF ~ '^[A-Za-z]+$' AND ALT ~ '^[A-Za-z]+$'
8057            """
8058        df_variants = self.get_query_to_df(query_variants)
8059
8060        # Added columns
8061        added_columns = []
8062
8063        # Add hgvs column in variants table
8064        hgvs_column_name = "hgvs_" + str(random.randrange(1000))
8065        added_column = self.add_column(
8066            table_variants, hgvs_column_name, "STRING", default_value=None
8067        )
8068        added_columns.append(added_column)
8069
8070        log.debug(f"refSeq loading...")
8071        # refSeq in duckDB
8072        refseq_table = get_refseq_table(
8073            conn=self.conn, refseq_table="refseq", refseq_file=refseq_file
8074        )
8075        # Loading all refSeq in Dataframe
8076        refseq_query = f"""
8077            SELECT df_variants.CHROM, df_variants.POS, {refseq_table}.name AS transcript
8078            FROM {refseq_table}
8079            JOIN df_variants ON (
8080                {refseq_table}.chrom = df_variants.CHROM
8081                AND {refseq_table}.txStart<=df_variants.POS
8082                AND {refseq_table}.txEnd>=df_variants.POS
8083            )
8084        """
8085        refseq_df = self.conn.query(refseq_query).pl()
8086
8087        if refseqlink_file:
8088            log.debug(f"refSeqLink loading...")
8089            # refSeqLink in duckDB
8090            refseqlink_table = get_refseq_table(
8091                conn=self.conn, refseq_table="refseqlink", refseq_file=refseqlink_file
8092            )
8093            # Loading all refSeqLink in Dataframe
8094            protacc_column = "protAcc_with_ver"
8095            mrnaacc_column = "mrnaAcc_with_ver"
8096            refseqlink_query = f"""
8097                SELECT {refseq_table}.chrom, {protacc_column} AS protein, {mrnaacc_column} AS transcript
8098                FROM {refseqlink_table} 
8099                JOIN {refseq_table} ON ({refseq_table}.name = {refseqlink_table}.mrnaAcc_with_ver)
8100                WHERE protAcc_without_ver IS NOT NULL
8101            """
8102            # Polars Dataframe
8103            refseqlink_df = self.conn.query(f"{refseqlink_query}").pl()
8104
8105        # Read RefSeq transcripts into a python dict/model.
8106        log.debug(f"Transcripts loading...")
8107        with tempfile.TemporaryDirectory() as tmpdir:
8108            transcripts_query = f"""
8109                COPY (
8110                    SELECT {refseq_table}.*
8111                    FROM {refseq_table}
8112                    JOIN df_variants ON (
8113                        {refseq_table}.chrom=df_variants.CHROM
8114                        AND {refseq_table}.txStart<=df_variants.POS
8115                        AND {refseq_table}.txEnd>=df_variants.POS
8116                    )
8117                )
8118                TO '{tmpdir}/transcript.tsv' (DELIMITER '\t');
8119            """
8120            self.conn.query(transcripts_query)
8121            with open(f"{tmpdir}/transcript.tsv") as infile:
8122                transcripts = read_transcripts(infile)
8123
8124        # Polars connexion
8125        polars_conn = pl.SQLContext(register_globals=True, eager=True)
8126
8127        log.debug("Genome loading...")
8128        # Read genome sequence using pyfaidx.
8129        genome = Fasta(genome_file)
8130
8131        log.debug("Start annotation HGVS...")
8132
8133        # Create
8134        # a Dask Dataframe from Pandas dataframe with partition as number of threads
8135        ddf = dd.from_pandas(df_variants, npartitions=threads)
8136
8137        # Use dask.dataframe.apply() to apply function on each partition
8138        ddf[hgvs_column_name] = ddf.map_partitions(partition_function)
8139
8140        # Convert Dask DataFrame to Pandas Dataframe
8141        df = ddf.compute()
8142
8143        # Convert Pandas dataframe to parquet (due to error in cast VARCHAR -> NULL ???)
8144        with tempfile.TemporaryDirectory() as tmpdir:
8145            df_parquet = os.path.join(tmpdir, "df.parquet")
8146            df.to_parquet(df_parquet)
8147
8148            # Update hgvs column
8149            update_variant_query = f"""
8150                UPDATE {table_variants}
8151                SET "{hgvs_column_name}"=df."{hgvs_column_name}"
8152                FROM read_parquet('{df_parquet}') as df
8153                WHERE variants."#CHROM" = df.CHROM
8154                AND variants.POS = df.POS
8155                AND variants.REF = df.REF
8156                AND variants.ALT = df.ALT
8157                AND df."{hgvs_column_name}" NOT IN ('') AND df."{hgvs_column_name}" NOT NULL
8158                """
8159            self.execute_query(update_variant_query)
8160
8161        # Update INFO column
8162        sql_query_update = f"""
8163            UPDATE {table_variants}
8164            SET INFO = 
8165                concat(
8166                    CASE 
8167                        WHEN INFO NOT IN ('','.')
8168                        THEN concat(INFO, ';')
8169                        ELSE ''
8170                    END,
8171                    'hgvs=',
8172                    {hgvs_column_name}
8173                )
8174            WHERE "{hgvs_column_name}" NOT IN ('') AND "{hgvs_column_name}" NOT NULL
8175            """
8176        self.execute_query(sql_query_update)
8177
8178        # Add header
8179        HGVS_INFOS = {
8180            "hgvs": {
8181                "ID": "hgvs",
8182                "Number": ".",
8183                "Type": "String",
8184                "Description": f"HGVS annotatation with HOWARD",
8185            }
8186        }
8187
8188        for field in HGVS_INFOS:
8189            field_ID = HGVS_INFOS[field]["ID"]
8190            field_description = HGVS_INFOS[field]["Description"]
8191            self.get_header().infos[field_ID] = vcf.parser._Info(
8192                field_ID,
8193                HGVS_INFOS[field]["Number"],
8194                HGVS_INFOS[field]["Type"],
8195                field_description,
8196                "unknown",
8197                "unknown",
8198                code_type_map[HGVS_INFOS[field]["Type"]],
8199            )
8200
8201        # Remove added columns
8202        for added_column in added_columns:
8203            self.drop_column(column=added_column)

The annotation_hgvs function performs HGVS annotation on a set of variants using genomic coordinates and alleles.

Parameters

threads: The threads parameter is an optional integer that specifies the number of threads to use for parallel processing. If no value is provided, it will default to the number of threads obtained from the get_threads() method

def get_operations_help( self, operations_config_dict: dict = {}, operations_config_file: str = None) -> list: View Source

8209    def get_operations_help(
8210        self, operations_config_dict: dict = {}, operations_config_file: str = None
8211    ) -> list:
8212
8213        # Init
8214        operations_help = []
8215
8216        # operations
8217        operations = self.get_config_json(
8218            name="calculations",
8219            config_dict=operations_config_dict,
8220            config_file=operations_config_file,
8221        )
8222        for op in operations:
8223            op_name = operations[op].get("name", op).upper()
8224            op_description = operations[op].get("description", op_name)
8225            op_available = operations[op].get("available", False)
8226            if op_available:
8227                operations_help.append(f"   {op_name}: {op_description}")
8228
8229        # Sort operations
8230        operations_help.sort()
8231
8232        # insert header
8233        operations_help.insert(0, "Available calculation operations:")
8234
8235        # Return
8236        return operations_help

def calculation( self, operations: dict = {}, operations_config_dict: dict = {}, operations_config_file: str = None) -> None: View Source

8238    def calculation(
8239        self,
8240        operations: dict = {},
8241        operations_config_dict: dict = {},
8242        operations_config_file: str = None,
8243    ) -> None:
8244        """
8245        It takes a list of operations, and for each operation, it checks if it's a python or sql
8246        operation, and then calls the appropriate function
8247
8248        param json example:
8249            "calculation": {
8250                "NOMEN": {
8251                    "options": {
8252                        "hgvs_field": "hgvs"
8253                    },
8254                "middle" : null
8255            }
8256        """
8257
8258        # Param
8259        param = self.get_param()
8260
8261        # CHeck operations config file
8262        if operations_config_file is None:
8263            operations_config_file = param.get("calculation", {}).get(
8264                "calculation_config", None
8265            )
8266
8267        # operations config
8268        operations_config = self.get_config_json(
8269            name="calculations",
8270            config_dict=operations_config_dict,
8271            config_file=operations_config_file,
8272        )
8273
8274        # Upper keys
8275        operations_config = {k.upper(): v for k, v in operations_config.items()}
8276
8277        # Calculations
8278
8279        # Operations from param
8280        operations = param.get("calculation", {}).get("calculations", operations)
8281
8282        # Quick calculation - add
8283        if param.get("calculations", None):
8284
8285            # List of operations
8286            calculations_list = [
8287                value.strip() for value in param.get("calculations", "").split(",")
8288            ]
8289
8290            # Log
8291            log.info(f"Quick Calculations:")
8292            for calculation_key in calculations_list:
8293                log.info(f"   {calculation_key}")
8294
8295            # Create tmp operations (to keep operation order)
8296            operations_tmp = {}
8297            for calculation_operation in calculations_list:
8298                if calculation_operation.upper() not in operations_tmp:
8299                    log.debug(
8300                        f"{calculation_operation}.upper() not in {operations_tmp}"
8301                    )
8302                    operations_tmp[calculation_operation.upper()] = {}
8303                    add_value_into_dict(
8304                        dict_tree=operations_tmp,
8305                        sections=[
8306                            calculation_operation.upper(),
8307                        ],
8308                        value=operations.get(calculation_operation.upper(), {}),
8309                    )
8310            # Add operations already in param
8311            for calculation_operation in operations:
8312                if calculation_operation not in operations_tmp:
8313                    operations_tmp[calculation_operation] = operations.get(
8314                        calculation_operation, {}
8315                    )
8316
8317            # Update operations in param
8318            operations = operations_tmp
8319
8320        # Operations for calculation
8321        if not operations:
8322            operations = param.get("calculation", {}).get("calculations", {})
8323
8324        if operations:
8325            log.info(f"Calculations...")
8326
8327        # For each operations
8328        for operation_name in operations:
8329            operation_name = operation_name.upper()
8330            if operation_name not in [""]:
8331                if operation_name in operations_config:
8332                    log.info(f"Calculation '{operation_name}'")
8333                    operation = operations_config[operation_name]
8334                    operation_type = operation.get("type", "sql")
8335                    if operation_type == "python":
8336                        self.calculation_process_function(
8337                            operation=operation, operation_name=operation_name
8338                        )
8339                    elif operation_type == "sql":
8340                        self.calculation_process_sql(
8341                            operation=operation, operation_name=operation_name
8342                        )
8343                    else:
8344                        log.error(
8345                            f"Operations config: Type '{operation_type}' NOT available"
8346                        )
8347                        raise ValueError(
8348                            f"Operations config: Type '{operation_type}' NOT available"
8349                        )
8350                else:
8351                    log.error(
8352                        f"Operations config: Calculation '{operation_name}' NOT available"
8353                    )
8354                    raise ValueError(
8355                        f"Operations config: Calculation '{operation_name}' NOT available"
8356                    )
8357
8358        # Explode INFOS fields into table fields
8359        if self.get_explode_infos():
8360            self.explode_infos(
8361                prefix=self.get_explode_infos_prefix(),
8362                fields=self.get_explode_infos_fields(),
8363                force=True,
8364            )

It takes a list of operations, and for each operation, it checks if it's a python or sql operation, and then calls the appropriate function

param json example: "calculation": { "NOMEN": { "options": { "hgvs_field": "hgvs" }, "middle" : null }

def calculation_process_sql(self, operation: dict, operation_name: str = 'unknown') -> None: View Source

8366    def calculation_process_sql(
8367        self, operation: dict, operation_name: str = "unknown"
8368    ) -> None:
8369        """
8370        The `calculation_process_sql` function takes in a mathematical operation as a string and
8371        performs the operation, updating the specified table with the result.
8372
8373        :param operation: The `operation` parameter is a dictionary that contains information about the
8374        mathematical operation to be performed. It includes the following keys:
8375        :type operation: dict
8376        :param operation_name: The `operation_name` parameter is a string that represents the name of
8377        the mathematical operation being performed. It is used for logging and error handling purposes,
8378        defaults to unknown
8379        :type operation_name: str (optional)
8380        """
8381
8382        # Operation infos
8383        operation_name = operation.get("name", "unknown")
8384        log.debug(f"process SQL {operation_name}")
8385        output_column_name = operation.get("output_column_name", operation_name)
8386        output_column_type = operation.get("output_column_type", "String")
8387        prefix = operation.get("explode_infos_prefix", "")
8388        output_column_type_sql = code_type_map_to_sql.get(output_column_type, "VARCHAR")
8389        output_column_description = operation.get(
8390            "output_column_description", f"{operation_name} operation"
8391        )
8392        operation_query = operation.get("operation_query", None)
8393        if isinstance(operation_query, list):
8394            operation_query = " ".join(operation_query)
8395        operation_info_fields = operation.get("info_fields", [])
8396        operation_info_fields_check = operation.get("info_fields_check", False)
8397        operation_info = operation.get("operation_info", True)
8398        operation_table = operation.get(
8399            "table", self.get_table_variants(clause="alter")
8400        )
8401
8402        # table variants
8403        if operation_table:
8404            table_variants = operation_table
8405        else:
8406            table_variants = self.get_table_variants(clause="alter")
8407
8408        if operation_query:
8409
8410            # Info fields check
8411            operation_info_fields_check_result = True
8412            if operation_info_fields_check:
8413                header_infos = self.get_header().infos
8414                for info_field in operation_info_fields:
8415                    operation_info_fields_check_result = (
8416                        operation_info_fields_check_result
8417                        and info_field in header_infos
8418                    )
8419
8420            # If info fields available
8421            if operation_info_fields_check_result:
8422
8423                # Added_columns
8424                added_columns = []
8425
8426                # Create VCF header field
8427                vcf_reader = self.get_header()
8428                vcf_reader.infos[output_column_name] = vcf.parser._Info(
8429                    output_column_name,
8430                    ".",
8431                    output_column_type,
8432                    output_column_description,
8433                    "howard calculation",
8434                    "0",
8435                    self.code_type_map.get(output_column_type),
8436                )
8437
8438                # Explode infos if needed
8439                log.debug(f"calculation_process_sql prefix {prefix}")
8440                added_columns += self.explode_infos(
8441                    prefix=prefix,
8442                    fields=[output_column_name] + operation_info_fields,
8443                    force=False,
8444                    table=table_variants,
8445                )
8446
8447                # Create column
8448                added_column = self.add_column(
8449                    table_name=table_variants,
8450                    column_name=prefix + output_column_name,
8451                    column_type=output_column_type_sql,
8452                    default_value="null",
8453                )
8454                added_columns.append(added_column)
8455
8456                # Operation calculation
8457                try:
8458
8459                    # Query to update calculation column
8460                    sql_update = f"""
8461                        UPDATE {table_variants}
8462                        SET "{prefix}{output_column_name}" = ({operation_query})
8463                    """
8464                    self.conn.execute(sql_update)
8465
8466                    # Add to INFO
8467                    if operation_info:
8468                        sql_update_info = f"""
8469                            UPDATE {table_variants}
8470                            SET "INFO" =
8471                                concat(
8472                                    CASE
8473                                        WHEN "INFO" IS NOT NULL
8474                                        THEN concat("INFO", ';')
8475                                        ELSE ''
8476                                    END,
8477                                    '{output_column_name}=',
8478                                    "{prefix}{output_column_name}"
8479                                )
8480                            WHERE "{prefix}{output_column_name}" IS NOT NULL AND "{prefix}{output_column_name}" NOT IN ('')
8481                        """
8482                        self.conn.execute(sql_update_info)
8483
8484                except:
8485                    log.error(
8486                        f"Operations config: Calculation '{operation_name}' query failed"
8487                    )
8488                    raise ValueError(
8489                        f"Operations config: Calculation '{operation_name}' query failed"
8490                    )
8491
8492                # Remove added columns
8493                for added_column in added_columns:
8494                    log.debug(f"added_column: {added_column}")
8495                    self.drop_column(column=added_column)
8496
8497            else:
8498                log.error(
8499                    f"Operations config: Calculation '{operation_name}' DOES NOT contain all mandatory fields {operation_info_fields}"
8500                )
8501                raise ValueError(
8502                    f"Operations config: Calculation '{operation_name}' DOES NOT contain all mandatory fields {operation_info_fields}"
8503                )
8504
8505        else:
8506            log.error(
8507                f"Operations config: Calculation '{operation_name}' query NOT defined"
8508            )
8509            raise ValueError(
8510                f"Operations config: Calculation '{operation_name}' query NOT defined"
8511            )

The calculation_process_sql function takes in a mathematical operation as a string and performs the operation, updating the specified table with the result.

Parameters

operation: The operation parameter is a dictionary that contains information about the mathematical operation to be performed. It includes the following keys:
operation_name: The operation_name parameter is a string that represents the name of the mathematical operation being performed. It is used for logging and error handling purposes, defaults to unknown

def calculation_process_function(self, operation: dict, operation_name: str = 'unknown') -> None: View Source

8513    def calculation_process_function(
8514        self, operation: dict, operation_name: str = "unknown"
8515    ) -> None:
8516        """
8517        The `calculation_process_function` takes in an operation dictionary and performs the specified
8518        function with the given parameters.
8519
8520        :param operation: The `operation` parameter is a dictionary that contains information about the
8521        operation to be performed. It has the following keys:
8522        :type operation: dict
8523        :param operation_name: The `operation_name` parameter is a string that represents the name of
8524        the operation being performed. It is used for logging purposes, defaults to unknown
8525        :type operation_name: str (optional)
8526        """
8527
8528        operation_name = operation["name"]
8529        log.debug(f"process Python {operation_name}")
8530        function_name = operation["function_name"]
8531        function_params = operation["function_params"]
8532        getattr(self, function_name)(*function_params)

The calculation_process_function takes in an operation dictionary and performs the specified function with the given parameters.

Parameters

operation: The operation parameter is a dictionary that contains information about the operation to be performed. It has the following keys:
operation_name: The operation_name parameter is a string that represents the name of the operation being performed. It is used for logging purposes, defaults to unknown

def calculation_variant_id(self) -> None: View Source

8534    def calculation_variant_id(self) -> None:
8535        """
8536        The function `calculation_variant_id` adds a variant ID annotation to a VCF file header and
8537        updates the INFO field of a variants table with the variant ID.
8538        """
8539
8540        # variant_id annotation field
8541        variant_id_tag = self.get_variant_id_column()
8542        added_columns = [variant_id_tag]
8543
8544        # variant_id hgvs tags"
8545        vcf_infos_tags = {
8546            variant_id_tag: "howard variant ID annotation",
8547        }
8548
8549        # Variants table
8550        table_variants = self.get_table_variants()
8551
8552        # Header
8553        vcf_reader = self.get_header()
8554
8555        # Add variant_id to header
8556        vcf_reader.infos[variant_id_tag] = vcf.parser._Info(
8557            variant_id_tag,
8558            ".",
8559            "String",
8560            vcf_infos_tags.get(variant_id_tag, "howard variant ID annotation"),
8561            "howard calculation",
8562            "0",
8563            self.code_type_map.get("String"),
8564        )
8565
8566        # Update
8567        sql_update = f"""
8568            UPDATE {table_variants}
8569            SET "INFO" = 
8570                concat(
8571                    CASE
8572                        WHEN "INFO" IS NULL OR "INFO" IN ('','.')
8573                        THEN ''
8574                        ELSE concat("INFO", ';')
8575                    END,
8576                    '{variant_id_tag}=',
8577                    "{variant_id_tag}"
8578                )
8579        """
8580        self.conn.execute(sql_update)
8581
8582        # Remove added columns
8583        for added_column in added_columns:
8584            self.drop_column(column=added_column)

The function calculation_variant_id adds a variant ID annotation to a VCF file header and updates the INFO field of a variants table with the variant ID.

def calculation_extract_snpeff_hgvs( self, snpeff_hgvs: str = 'snpeff_hgvs', snpeff_field: str = 'ANN') -> None: View Source

8586    def calculation_extract_snpeff_hgvs(
8587        self,
8588        snpeff_hgvs: str = "snpeff_hgvs",
8589        snpeff_field: str = "ANN",
8590    ) -> None:
8591        """
8592        The function `calculation_extract_snpeff_hgvs` extracts HGVS nomenclatures from the SnpEff
8593        annotation field in a VCF file and adds them as a new column in the variants table.
8594
8595        :param snpeff_hgvs: The `snpeff_hgvs` parameter in the `calculation_extract_snpeff_hgvs`
8596        function is used to specify the name of the column that will store the HGVS nomenclatures
8597        extracted from the SnpEff annotation field in a VCF file. This parameter allows you, defaults to
8598        snpeff_hgvs
8599        :type snpeff_hgvs: str (optional)
8600        :param snpeff_field: The `snpeff_field` parameter in the `calculation_extract_snpeff_hgvs`
8601        function represents the field in the VCF file that contains SnpEff annotations. This field is
8602        used to extract HGVS nomenclatures from the SnpEff annotation field and add them as a, defaults
8603        to ANN
8604        :type snpeff_field: str (optional)
8605        """
8606
8607        # Snpeff hgvs tags
8608        vcf_infos_tags = {
8609            snpeff_hgvs: "HGVS nomenclatures from snpEff annotation",
8610        }
8611
8612        # Prefix
8613        prefix = self.get_explode_infos_prefix()
8614        if prefix:
8615            prefix = "INFO/"
8616
8617        # snpEff fields
8618        speff_ann_infos = prefix + snpeff_field
8619        speff_hgvs_infos = prefix + snpeff_hgvs
8620
8621        # Variants table
8622        table_variants = self.get_table_variants()
8623
8624        # Header
8625        vcf_reader = self.get_header()
8626
8627        # Add columns
8628        added_columns = []
8629
8630        # Explode HGVS field in column
8631        added_columns += self.explode_infos(fields=[snpeff_field])
8632
8633        if snpeff_field in vcf_reader.infos:
8634
8635            log.debug(vcf_reader.infos[snpeff_field])
8636
8637            # Extract ANN header
8638            ann_description = vcf_reader.infos[snpeff_field].desc
8639            pattern = r"'(.+?)'"
8640            match = re.search(pattern, ann_description)
8641            if match:
8642                ann_header_match = match.group(1).split(" | ")
8643                ann_header_desc = {}
8644                for i in range(len(ann_header_match)):
8645                    ann_header_info = "".join(
8646                        char for char in ann_header_match[i] if char.isalnum()
8647                    )
8648                    ann_header_desc[ann_header_info] = ann_header_match[i]
8649                if not ann_header_desc:
8650                    raise ValueError("Invalid header description format")
8651            else:
8652                raise ValueError("Invalid header description format")
8653
8654            # Create variant id
8655            variant_id_column = self.get_variant_id_column()
8656            added_columns += [variant_id_column]
8657
8658            # Create dataframe
8659            dataframe_snpeff_hgvs = self.get_query_to_df(
8660                f""" SELECT "{variant_id_column}", "{speff_ann_infos}" FROM {table_variants} """
8661            )
8662
8663            # Create main NOMEN column
8664            dataframe_snpeff_hgvs[speff_hgvs_infos] = dataframe_snpeff_hgvs[
8665                speff_ann_infos
8666            ].apply(
8667                lambda x: extract_snpeff_hgvs(
8668                    str(x), header=list(ann_header_desc.values())
8669                )
8670            )
8671
8672            # Add snpeff_hgvs to header
8673            vcf_reader.infos[snpeff_hgvs] = vcf.parser._Info(
8674                snpeff_hgvs,
8675                ".",
8676                "String",
8677                vcf_infos_tags.get(snpeff_hgvs, "snpEff hgvs annotations"),
8678                "howard calculation",
8679                "0",
8680                self.code_type_map.get("String"),
8681            )
8682
8683            # Update
8684            sql_update = f"""
8685                UPDATE variants
8686                SET "INFO" = 
8687                    concat(
8688                        CASE
8689                            WHEN "INFO" IS NULL OR "INFO" IN ('','.')
8690                            THEN ''
8691                            ELSE concat("INFO", ';')
8692                        END,
8693                        CASE 
8694                            WHEN dataframe_snpeff_hgvs."{speff_hgvs_infos}" NOT IN ('','.','NaN')
8695                            AND dataframe_snpeff_hgvs."{speff_hgvs_infos}" NOT NULL
8696                            THEN concat(
8697                                    '{snpeff_hgvs}=',
8698                                    dataframe_snpeff_hgvs."{speff_hgvs_infos}"
8699                                )
8700                            ELSE ''
8701                        END
8702                    )
8703                FROM dataframe_snpeff_hgvs
8704                WHERE {table_variants}."{variant_id_column}" = dataframe_snpeff_hgvs."{variant_id_column}"
8705
8706            """
8707            self.conn.execute(sql_update)
8708
8709            # Delete dataframe
8710            del dataframe_snpeff_hgvs
8711            gc.collect()
8712
8713        else:
8714
8715            log.warning(
8716                "No snpEff annotation. Please Anotate with snpEff before use this calculation option"
8717            )
8718
8719        # Remove added columns
8720        for added_column in added_columns:
8721            self.drop_column(column=added_column)

The function calculation_extract_snpeff_hgvs extracts HGVS nomenclatures from the SnpEff annotation field in a VCF file and adds them as a new column in the variants table.

Parameters

snpeff_hgvs: The snpeff_hgvs parameter in the calculation_extract_snpeff_hgvs function is used to specify the name of the column that will store the HGVS nomenclatures extracted from the SnpEff annotation field in a VCF file. This parameter allows you, defaults to snpeff_hgvs
snpeff_field: The snpeff_field parameter in the calculation_extract_snpeff_hgvs function represents the field in the VCF file that contains SnpEff annotations. This field is used to extract HGVS nomenclatures from the SnpEff annotation field and add them as a, defaults to ANN

def calculation_snpeff_ann_explode( self, uniquify: bool = True, output_format: str = 'fields', output_prefix: str = 'snpeff_', snpeff_field: str = 'ANN') -> None: View Source

8723    def calculation_snpeff_ann_explode(
8724        self,
8725        uniquify: bool = True,
8726        output_format: str = "fields",
8727        output_prefix: str = "snpeff_",
8728        snpeff_field: str = "ANN",
8729    ) -> None:
8730        """
8731        The `calculation_snpeff_ann_explode` function processes SnpEff annotations in a VCF file by
8732        exploding the HGVS field and updating variant information accordingly.
8733
8734        :param uniquify: The `uniquify` parameter in the `calculation_snpeff_ann_explode` method is a
8735        boolean flag that determines whether the output should be uniquified or not. When set to `True`,
8736        it indicates that the output should be unique, meaning that duplicate entries should be removed,
8737        defaults to True
8738        :type uniquify: bool (optional)
8739        :param output_format: The `output_format` parameter in the `calculation_snpeff_ann_explode`
8740        function specifies the format in which the output annotations will be generated. It has a
8741        default value of "fields". You can also set it to "JSON" to output the annotations in JSON
8742        format, defaults to fields
8743        :type output_format: str (optional)
8744        :param output_prefix: The `output_prefix` parameter in the `calculation_snpeff_ann_explode`
8745        method is used to specify the prefix that will be added to the output annotations generated
8746        during the calculation process. This prefix helps to differentiate the newly added annotations
8747        from existing ones in the output data. By default, the, defaults to ANN_
8748        :type output_prefix: str (optional)
8749        :param snpeff_field: The `snpeff_field` parameter in the `calculation_snpeff_ann_explode`
8750        function is used to specify the field in the VCF file that contains SnpEff annotations. This
8751        field will be processed to explode the HGVS annotations and update the variant information
8752        accordingly, defaults to ANN
8753        :type snpeff_field: str (optional)
8754        """
8755
8756        # SnpEff annotation field
8757        snpeff_hgvs = "snpeff_ann_explode"
8758
8759        # Snpeff hgvs tags
8760        vcf_infos_tags = {
8761            snpeff_hgvs: "Explode snpEff annotations",
8762        }
8763
8764        # Prefix
8765        prefix = self.get_explode_infos_prefix()
8766        if prefix:
8767            prefix = "INFO/"
8768
8769        # snpEff fields
8770        speff_ann_infos = prefix + snpeff_field
8771        speff_hgvs_infos = prefix + snpeff_hgvs
8772
8773        # Variants table
8774        table_variants = self.get_table_variants()
8775
8776        # Header
8777        vcf_reader = self.get_header()
8778
8779        # Add columns
8780        added_columns = []
8781
8782        # Explode HGVS field in column
8783        added_columns += self.explode_infos(fields=[snpeff_field])
8784        log.debug(f"snpeff_field={snpeff_field}")
8785        log.debug(f"added_columns={added_columns}")
8786
8787        if snpeff_field in vcf_reader.infos:
8788
8789            # Extract ANN header
8790            ann_description = vcf_reader.infos[snpeff_field].desc
8791            pattern = r"'(.+?)'"
8792            match = re.search(pattern, ann_description)
8793            if match:
8794                ann_header_match = match.group(1).split(" | ")
8795                ann_header = []
8796                ann_header_desc = {}
8797                for i in range(len(ann_header_match)):
8798                    ann_header_info = "".join(
8799                        char for char in ann_header_match[i] if char.isalnum()
8800                    )
8801                    ann_header.append(ann_header_info)
8802                    ann_header_desc[ann_header_info] = ann_header_match[i]
8803                if not ann_header_desc:
8804                    raise ValueError("Invalid header description format")
8805            else:
8806                raise ValueError("Invalid header description format")
8807
8808            # Create variant id
8809            variant_id_column = self.get_variant_id_column()
8810            added_columns += [variant_id_column]
8811
8812            # Create dataframe
8813            dataframe_snpeff_hgvs = self.get_query_to_df(
8814                f""" SELECT "{variant_id_column}", "{speff_ann_infos}" FROM {table_variants} """
8815            )
8816
8817            # Create snpEff columns
8818            dataframe_snpeff_hgvs[speff_hgvs_infos] = dataframe_snpeff_hgvs[
8819                speff_ann_infos
8820            ].apply(
8821                lambda x: explode_snpeff_ann(
8822                    str(x),
8823                    uniquify=uniquify,
8824                    output_format=output_format,
8825                    prefix=output_prefix,
8826                    header=list(ann_header_desc.values()),
8827                )
8828            )
8829
8830            # Header
8831            ann_annotations_prefix = ""
8832            if output_format.upper() in ["JSON"]:
8833                ann_annotations_prefix = f"{output_prefix}="
8834                vcf_reader.infos[output_prefix] = vcf.parser._Info(
8835                    output_prefix,
8836                    ".",
8837                    "String",
8838                    vcf_infos_tags.get(snpeff_hgvs, "snpEff annotations")
8839                    + " - JSON format",
8840                    "howard calculation",
8841                    "0",
8842                    self.code_type_map.get("String"),
8843                )
8844            else:
8845                for ann_annotation in ann_header:
8846                    ann_annotation_id = f"{output_prefix}{ann_annotation}"
8847                    vcf_reader.infos[ann_annotation_id] = vcf.parser._Info(
8848                        ann_annotation_id,
8849                        ".",
8850                        "String",
8851                        vcf_infos_tags.get(snpeff_hgvs, "snpEff annotations")
8852                        + f" - '{ann_header_desc[ann_annotation]}' annotation",
8853                        "howard calculation",
8854                        "0",
8855                        self.code_type_map.get("String"),
8856                    )
8857
8858            # Update
8859            sql_update = f"""
8860                UPDATE variants
8861                SET "INFO" = 
8862                    concat(
8863                        CASE
8864                            WHEN "INFO" IS NULL OR "INFO" IN ('','.')
8865                            THEN ''
8866                            ELSE concat("INFO", ';')
8867                        END,
8868                        CASE 
8869                            WHEN dataframe_snpeff_hgvs."{speff_hgvs_infos}" NOT IN ('','.','NaN')
8870                                AND dataframe_snpeff_hgvs."{speff_hgvs_infos}" NOT NULL
8871                            THEN concat(
8872                                '{ann_annotations_prefix}',
8873                                dataframe_snpeff_hgvs."{speff_hgvs_infos}"
8874                                )
8875                            ELSE ''
8876                        END
8877                    )
8878                FROM dataframe_snpeff_hgvs
8879                WHERE {table_variants}."{variant_id_column}" = dataframe_snpeff_hgvs."{variant_id_column}"
8880
8881            """
8882            self.conn.execute(sql_update)
8883
8884            # Delete dataframe
8885            del dataframe_snpeff_hgvs
8886            gc.collect()
8887
8888        else:
8889
8890            log.warning(
8891                "No snpEff annotation. Please Anotate with snpEff before use this calculation option"
8892            )
8893
8894        # Remove added columns
8895        for added_column in added_columns:
8896            self.drop_column(column=added_column)

The calculation_snpeff_ann_explode function processes SnpEff annotations in a VCF file by exploding the HGVS field and updating variant information accordingly.

Parameters

uniquify: The uniquify parameter in the calculation_snpeff_ann_explode method is a boolean flag that determines whether the output should be uniquified or not. When set to True, it indicates that the output should be unique, meaning that duplicate entries should be removed, defaults to True
output_format: The output_format parameter in the calculation_snpeff_ann_explode function specifies the format in which the output annotations will be generated. It has a default value of "fields". You can also set it to "JSON" to output the annotations in JSON format, defaults to fields
output_prefix: The output_prefix parameter in the calculation_snpeff_ann_explode method is used to specify the prefix that will be added to the output annotations generated during the calculation process. This prefix helps to differentiate the newly added annotations from existing ones in the output data. By default, the, defaults to ANN_
snpeff_field: The snpeff_field parameter in the calculation_snpeff_ann_explode function is used to specify the field in the VCF file that contains SnpEff annotations. This field will be processed to explode the HGVS annotations and update the variant information accordingly, defaults to ANN

def calculation_extract_nomen(self) -> None: View Source

8898    def calculation_extract_nomen(self) -> None:
8899        """
8900        This function extracts the HGVS nomenclature from the calculation/identification of NOMEN.
8901        """
8902
8903        # NOMEN field
8904        field_nomen_dict = "NOMEN_DICT"
8905
8906        # NOMEN structure
8907        nomen_dict = {
8908            "NOMEN": "NOMEN hgvs nomenclature considered as reference hgvs (official transcript, first otherwise)",
8909            "CNOMEN": "CNOMEN hgvs nomenclature at DNA level related to a transcript (TNOMEN)",
8910            "RNOMEN": "RNOMEN hgvs nomenclature at RNA level related to a transcript (TNOMEN)",
8911            "NNOMEN": "NNOMEN hgvs nomenclature for non-coding variant",
8912            "PNOMEN": "PNOMEN hgvs nomenclature at Protein level related to a transcript (TNOMEN)",
8913            "TVNOMEN": "TVNOMEN hgvs transcript with version (if any) used (e.g. for CNOMEN and PNOMEN)",
8914            "TNOMEN": "TNOMEN hgvs transcript used (e.g. for CNOMEN and PNOMEN)",
8915            "VNOMEN": "VNOMEN hgvs transcript version used (e.g. for CNOMEN and PNOMEN)",
8916            "ENOMEN": "ENOMEN hgvs exon nomenclature related to a transcript (TNOMEN)",
8917            "GNOMEN": "GNOMEN hgvs gene nomenclature related to a transcript (TNOMEN)",
8918        }
8919
8920        # Param
8921        param = self.get_param()
8922
8923        # Threads
8924        threads = self.get_threads()
8925
8926        # Prefix
8927        prefix = self.get_explode_infos_prefix()
8928
8929        # Header
8930        vcf_reader = self.get_header()
8931
8932        # Added columns
8933        added_columns = []
8934
8935        # Get HGVS field
8936        hgvs_field = (
8937            param.get("calculation", {})
8938            .get("calculations", {})
8939            .get("NOMEN", {})
8940            .get("options", {})
8941            .get("hgvs_field", "hgvs")
8942        )
8943
8944        # Get NOMEN pattern
8945        nomen_pattern = (
8946            param.get("calculation", {})
8947            .get("calculations", {})
8948            .get("NOMEN", {})
8949            .get("options", {})
8950            .get("pattern", None)
8951        )
8952
8953        # transcripts list of preference sources
8954        transcripts_sources = {}
8955
8956        # Get transcripts
8957        transcripts_file = (
8958            param.get("calculation", {})
8959            .get("calculations", {})
8960            .get("NOMEN", {})
8961            .get("options", {})
8962            .get("transcripts", None)
8963        )
8964        transcripts_file = full_path(transcripts_file)
8965        if transcripts_file:
8966            if os.path.exists(transcripts_file):
8967                transcripts_dataframe = transcripts_file_to_df(transcripts_file)
8968                transcripts_from_file = transcripts_dataframe.iloc[:, 0].tolist()
8969                transcripts_sources["file"] = transcripts_from_file
8970            else:
8971                msg_err = f"Transcript file '{transcripts_file}' does NOT exist"
8972                log.error(msg_err)
8973                raise ValueError(msg_err)
8974
8975        # Get transcripts table
8976        transcripts_table = (
8977            param.get("calculation", {})
8978            .get("calculations", {})
8979            .get("NOMEN", {})
8980            .get("options", {})
8981            .get("transcripts_table", self.get_table_variants())
8982        )
8983        # Get transcripts column
8984        transcripts_column = (
8985            param.get("calculation", {})
8986            .get("calculations", {})
8987            .get("NOMEN", {})
8988            .get("options", {})
8989            .get("transcripts_column", None)
8990        )
8991
8992        if transcripts_table and transcripts_column:
8993            extra_field_transcript = f"{transcripts_table}.{transcripts_column}"
8994            # Explode if not exists
8995            added_columns += self.explode_infos(
8996                fields=[transcripts_column], table=transcripts_table
8997            )
8998        else:
8999            extra_field_transcript = f"NULL"
9000
9001        # Transcripts of preference source order
9002        transcripts_order = (
9003            param.get("calculation", {})
9004            .get("calculations", {})
9005            .get("NOMEN", {})
9006            .get("options", {})
9007            .get("transcripts_order", ["column", "file"])
9008        )
9009
9010        # Transcripts from file
9011        transcripts = transcripts_sources.get("file", [])
9012
9013        # Explode HGVS field in column
9014        added_columns += self.explode_infos(fields=[hgvs_field])
9015
9016        # extra infos
9017        extra_infos = self.get_extra_infos()
9018        extra_field = prefix + hgvs_field
9019
9020        if extra_field in extra_infos:
9021
9022            # Create dataframe
9023            dataframe_hgvs = self.get_query_to_df(
9024                f""" SELECT "#CHROM", "POS", "REF", "ALT", "{extra_field}" AS hgvs, {extra_field_transcript} AS transcript FROM variants """
9025            )
9026
9027            # Transcripts rank
9028            transcripts_rank = {
9029                transcript: rank for rank, transcript in enumerate(transcripts, start=1)
9030            }
9031            transcripts_len = len(transcripts_rank)
9032
9033            # Create main NOMEN column
9034            dataframe_hgvs[field_nomen_dict] = dataframe_hgvs.apply(
9035                lambda x: find_nomen(
9036                    hgvs=x.hgvs,
9037                    transcript=x.transcript,
9038                    transcripts=transcripts_rank,
9039                    pattern=nomen_pattern,
9040                    transcripts_source_order=transcripts_order,
9041                    transcripts_len=transcripts_len,
9042                ),
9043                axis=1,
9044            )
9045
9046            # Explode NOMEN Structure and create SQL set for update
9047            sql_nomen_fields = []
9048            for nomen_field in nomen_dict:
9049
9050                # Create VCF header field
9051                vcf_reader.infos[nomen_field] = vcf.parser._Info(
9052                    nomen_field,
9053                    ".",
9054                    "String",
9055                    nomen_dict.get(nomen_field, "howard calculation NOMEN"),
9056                    "howard calculation",
9057                    "0",
9058                    self.code_type_map.get("String"),
9059                )
9060
9061                # Add field to SQL query update
9062                sql_nomen_fields.append(
9063                    f"""
9064                        CASE 
9065                            WHEN dataframe_hgvs."{field_nomen_dict}"."{nomen_field}" NOT NULL AND dataframe_hgvs."{field_nomen_dict}"."{nomen_field}" NOT IN ('')
9066                            THEN concat(
9067                                    ';{nomen_field}=',
9068                                    dataframe_hgvs."{field_nomen_dict}"."{nomen_field}"
9069                                )
9070                            ELSE ''
9071                        END
9072                    """
9073                )
9074
9075            # SQL set for update
9076            sql_nomen_fields_set = ", ".join(sql_nomen_fields)
9077
9078            # Update
9079            sql_update = f"""
9080                UPDATE variants
9081                SET "INFO" = 
9082                    concat(
9083                        CASE
9084                            WHEN "INFO" IS NULL
9085                            THEN ''
9086                            ELSE "INFO"
9087                        END,
9088                        {sql_nomen_fields_set}
9089                    )
9090                FROM dataframe_hgvs
9091                WHERE variants."#CHROM" = dataframe_hgvs."#CHROM"
9092                    AND variants."POS" = dataframe_hgvs."POS" 
9093                    AND variants."REF" = dataframe_hgvs."REF"
9094                    AND variants."ALT" = dataframe_hgvs."ALT"
9095            """
9096            self.conn.execute(sql_update)
9097
9098            # Delete dataframe
9099            del dataframe_hgvs
9100            gc.collect()
9101
9102        # Remove added columns
9103        for added_column in added_columns:
9104            self.drop_column(column=added_column)

This function extracts the HGVS nomenclature from the calculation/identification of NOMEN.

def calculation_find_by_pipeline(self, tag: str = 'findbypipeline') -> None: View Source

9106    def calculation_find_by_pipeline(self, tag: str = "findbypipeline") -> None:
9107        """
9108        The function `calculation_find_by_pipeline` performs a calculation to find the number of
9109        pipeline/sample for a variant and updates the variant information in a VCF file.
9110
9111        :param tag: The `tag` parameter is a string that represents the annotation field for the
9112        "findbypipeline" information in the VCF file. It is used to create the annotation field in the
9113        VCF header and to update the corresponding field in the variants table, defaults to
9114        findbypipeline
9115        :type tag: str (optional)
9116        """
9117
9118        # if FORMAT and samples
9119        if (
9120            "FORMAT" in self.get_header_columns_as_list()
9121            and self.get_header_sample_list()
9122        ):
9123
9124            # findbypipeline annotation field
9125            findbypipeline_tag = tag
9126
9127            # VCF infos tags
9128            vcf_infos_tags = {
9129                findbypipeline_tag: f"Number of pipeline/sample for a variant ({findbypipeline_tag})",
9130            }
9131
9132            # Prefix
9133            prefix = self.get_explode_infos_prefix()
9134
9135            # Field
9136            findbypipeline_infos = prefix + findbypipeline_tag
9137
9138            # Variants table
9139            table_variants = self.get_table_variants()
9140
9141            # Header
9142            vcf_reader = self.get_header()
9143
9144            # Create variant id
9145            variant_id_column = self.get_variant_id_column()
9146            added_columns = [variant_id_column]
9147
9148            # variant_id, FORMAT and samples
9149            samples_fields = f" {variant_id_column}, FORMAT , " + " , ".join(
9150                [f""" "{sample}" """ for sample in self.get_header_sample_list()]
9151            )
9152
9153            # Create dataframe
9154            dataframe_findbypipeline = self.get_query_to_df(
9155                f""" SELECT {samples_fields} FROM {table_variants} """
9156            )
9157
9158            # Create findbypipeline column
9159            dataframe_findbypipeline[findbypipeline_infos] = (
9160                dataframe_findbypipeline.apply(
9161                    lambda row: findbypipeline(
9162                        row, samples=self.get_header_sample_list()
9163                    ),
9164                    axis=1,
9165                )
9166            )
9167
9168            # Add snpeff_hgvs to header
9169            vcf_reader.infos[findbypipeline_tag] = vcf.parser._Info(
9170                findbypipeline_tag,
9171                ".",
9172                "String",
9173                vcf_infos_tags.get(findbypipeline_tag, "Find in pipeline/sample"),
9174                "howard calculation",
9175                "0",
9176                self.code_type_map.get("String"),
9177            )
9178
9179            # Update
9180            sql_update = f"""
9181                UPDATE variants
9182                SET "INFO" = 
9183                    concat(
9184                        CASE
9185                            WHEN "INFO" IS NULL OR "INFO" IN ('','.')
9186                            THEN ''
9187                            ELSE concat("INFO", ';')
9188                        END,
9189                        CASE 
9190                            WHEN dataframe_findbypipeline."{findbypipeline_infos}" NOT IN ('','.')
9191                                AND dataframe_findbypipeline."{findbypipeline_infos}" NOT NULL
9192                            THEN concat(
9193                                    '{findbypipeline_tag}=',
9194                                    dataframe_findbypipeline."{findbypipeline_infos}"
9195                                )
9196                            ELSE ''
9197                        END
9198                    )
9199                FROM dataframe_findbypipeline
9200                WHERE variants."{variant_id_column}" = dataframe_findbypipeline."{variant_id_column}"
9201            """
9202            self.conn.execute(sql_update)
9203
9204            # Remove added columns
9205            for added_column in added_columns:
9206                self.drop_column(column=added_column)
9207
9208            # Delete dataframe
9209            del dataframe_findbypipeline
9210            gc.collect()

The function calculation_find_by_pipeline performs a calculation to find the number of pipeline/sample for a variant and updates the variant information in a VCF file.

Parameters

tag: The tag parameter is a string that represents the annotation field for the "findbypipeline" information in the VCF file. It is used to create the annotation field in the VCF header and to update the corresponding field in the variants table, defaults to findbypipeline

def calculation_genotype_concordance(self) -> None: View Source

9212    def calculation_genotype_concordance(self) -> None:
9213        """
9214        The function `calculation_genotype_concordance` calculates the genotype concordance for
9215        multi-caller VCF files and updates the variant information in the database.
9216        """
9217
9218        # if FORMAT and samples
9219        if (
9220            "FORMAT" in self.get_header_columns_as_list()
9221            and self.get_header_sample_list()
9222        ):
9223
9224            # genotypeconcordance annotation field
9225            genotypeconcordance_tag = "genotypeconcordance"
9226
9227            # VCF infos tags
9228            vcf_infos_tags = {
9229                genotypeconcordance_tag: "Concordance of genotype for multi caller VCF",
9230            }
9231
9232            # Prefix
9233            prefix = self.get_explode_infos_prefix()
9234
9235            # Field
9236            genotypeconcordance_infos = prefix + genotypeconcordance_tag
9237
9238            # Variants table
9239            table_variants = self.get_table_variants()
9240
9241            # Header
9242            vcf_reader = self.get_header()
9243
9244            # Create variant id
9245            variant_id_column = self.get_variant_id_column()
9246            added_columns = [variant_id_column]
9247
9248            # variant_id, FORMAT and samples
9249            samples_fields = f" {variant_id_column}, FORMAT , " + " , ".join(
9250                [f""" "{sample}" """ for sample in self.get_header_sample_list()]
9251            )
9252
9253            # Create dataframe
9254            dataframe_genotypeconcordance = self.get_query_to_df(
9255                f""" SELECT {samples_fields} FROM {table_variants} """
9256            )
9257
9258            # Create genotypeconcordance column
9259            dataframe_genotypeconcordance[genotypeconcordance_infos] = (
9260                dataframe_genotypeconcordance.apply(
9261                    lambda row: genotypeconcordance(
9262                        row, samples=self.get_header_sample_list()
9263                    ),
9264                    axis=1,
9265                )
9266            )
9267
9268            # Add genotypeconcordance to header
9269            vcf_reader.infos[genotypeconcordance_tag] = vcf.parser._Info(
9270                genotypeconcordance_tag,
9271                ".",
9272                "String",
9273                vcf_infos_tags.get(genotypeconcordance_tag, "snpEff hgvs annotations"),
9274                "howard calculation",
9275                "0",
9276                self.code_type_map.get("String"),
9277            )
9278
9279            # Update
9280            sql_update = f"""
9281                UPDATE variants
9282                SET "INFO" = 
9283                    concat(
9284                        CASE
9285                            WHEN "INFO" IS NULL OR "INFO" IN ('','.')
9286                            THEN ''
9287                            ELSE concat("INFO", ';')
9288                        END,
9289                        CASE
9290                            WHEN dataframe_genotypeconcordance."{genotypeconcordance_infos}" NOT IN ('','.')
9291                                AND dataframe_genotypeconcordance."{genotypeconcordance_infos}" NOT NULL
9292                            THEN concat(
9293                                    '{genotypeconcordance_tag}=',
9294                                    dataframe_genotypeconcordance."{genotypeconcordance_infos}"
9295                                )
9296                            ELSE ''
9297                        END
9298                    )
9299                FROM dataframe_genotypeconcordance
9300                WHERE variants."{variant_id_column}" = dataframe_genotypeconcordance."{variant_id_column}"
9301            """
9302            self.conn.execute(sql_update)
9303
9304            # Remove added columns
9305            for added_column in added_columns:
9306                self.drop_column(column=added_column)
9307
9308            # Delete dataframe
9309            del dataframe_genotypeconcordance
9310            gc.collect()

The function calculation_genotype_concordance calculates the genotype concordance for multi-caller VCF files and updates the variant information in the database.

def calculation_barcode(self, tag: str = 'barcode') -> None: View Source

9312    def calculation_barcode(self, tag: str = "barcode") -> None:
9313        """
9314        The `calculation_barcode` function calculates barcode values for variants in a VCF file and
9315        updates the INFO field in the file with the calculated barcode values.
9316
9317        :param tag: The `tag` parameter in the `calculation_barcode` function is used to specify the tag
9318        name that will be used for the barcode calculation in the VCF file. If no tag name is provided,
9319        the default tag name is set to "barcode", defaults to barcode
9320        :type tag: str (optional)
9321        """
9322
9323        # if FORMAT and samples
9324        if (
9325            "FORMAT" in self.get_header_columns_as_list()
9326            and self.get_header_sample_list()
9327        ):
9328
9329            # barcode annotation field
9330            if not tag:
9331                tag = "barcode"
9332
9333            # VCF infos tags
9334            vcf_infos_tags = {
9335                tag: "barcode calculation (VaRank)",
9336            }
9337
9338            # Prefix
9339            prefix = self.get_explode_infos_prefix()
9340
9341            # Field
9342            barcode_infos = prefix + tag
9343
9344            # Variants table
9345            table_variants = self.get_table_variants()
9346
9347            # Header
9348            vcf_reader = self.get_header()
9349
9350            # Create variant id
9351            variant_id_column = self.get_variant_id_column()
9352            added_columns = [variant_id_column]
9353
9354            # variant_id, FORMAT and samples
9355            samples_fields = f" {variant_id_column}, FORMAT , " + " , ".join(
9356                [f""" "{sample}" """ for sample in self.get_header_sample_list()]
9357            )
9358
9359            # Create dataframe
9360            dataframe_barcode = self.get_query_to_df(
9361                f""" SELECT {samples_fields} FROM {table_variants} """
9362            )
9363
9364            # Create barcode column
9365            dataframe_barcode[barcode_infos] = dataframe_barcode.apply(
9366                lambda row: barcode(row, samples=self.get_header_sample_list()), axis=1
9367            )
9368
9369            # Add barcode to header
9370            vcf_reader.infos[tag] = vcf.parser._Info(
9371                tag,
9372                ".",
9373                "String",
9374                vcf_infos_tags.get(tag, vcf_infos_tags.get(tag)),
9375                "howard calculation",
9376                "0",
9377                self.code_type_map.get("String"),
9378            )
9379
9380            # Update
9381            sql_update = f"""
9382                UPDATE {table_variants}
9383                SET "INFO" = 
9384                    concat(
9385                        CASE
9386                            WHEN "INFO" IS NULL OR "INFO" IN ('','.')
9387                            THEN ''
9388                            ELSE concat("INFO", ';')
9389                        END,
9390                        CASE
9391                            WHEN dataframe_barcode."{barcode_infos}" NOT IN ('','.')
9392                            AND dataframe_barcode."{barcode_infos}" NOT NULL
9393                            THEN concat(
9394                                    '{tag}=',
9395                                    dataframe_barcode."{barcode_infos}"
9396                                )
9397                            ELSE ''
9398                        END
9399                    )
9400                FROM dataframe_barcode
9401                WHERE {table_variants}."{variant_id_column}" = dataframe_barcode."{variant_id_column}"
9402            """
9403            self.conn.execute(sql_update)
9404
9405            # Remove added columns
9406            for added_column in added_columns:
9407                self.drop_column(column=added_column)
9408
9409            # Delete dataframe
9410            del dataframe_barcode
9411            gc.collect()

The calculation_barcode function calculates barcode values for variants in a VCF file and updates the INFO field in the file with the calculated barcode values.

Parameters

tag: The tag parameter in the calculation_barcode function is used to specify the tag name that will be used for the barcode calculation in the VCF file. If no tag name is provided, the default tag name is set to "barcode", defaults to barcode

def calculation_barcode_family(self, tag: str = 'BCF') -> None: View Source

9413    def calculation_barcode_family(self, tag: str = "BCF") -> None:
9414        """
9415        The `calculation_barcode_family` function calculates barcode values for variants in a VCF file
9416        and updates the INFO field in the file with the calculated barcode values.
9417
9418        :param tag: The `tag` parameter in the `calculation_barcode_family` function is used to specify
9419        the barcode tag that will be added to the VCF file during the calculation process. If no value
9420        is provided for the `tag` parameter, the default value used is "BCF", defaults to BCF
9421        :type tag: str (optional)
9422        """
9423
9424        # if FORMAT and samples
9425        if (
9426            "FORMAT" in self.get_header_columns_as_list()
9427            and self.get_header_sample_list()
9428        ):
9429
9430            # barcode annotation field
9431            if not tag:
9432                tag = "BCF"
9433
9434            # VCF infos tags
9435            vcf_infos_tags = {
9436                tag: "barcode family calculation",
9437                f"{tag}S": "barcode family samples",
9438            }
9439
9440            # Param
9441            param = self.get_param()
9442            log.debug(f"param={param}")
9443
9444            # Prefix
9445            prefix = self.get_explode_infos_prefix()
9446
9447            # PED param
9448            ped = (
9449                param.get("calculation", {})
9450                .get("calculations", {})
9451                .get("BARCODEFAMILY", {})
9452                .get("family_pedigree", None)
9453            )
9454            log.debug(f"ped={ped}")
9455
9456            # Load PED
9457            if ped:
9458
9459                # Pedigree is a file
9460                if isinstance(ped, str) and os.path.exists(full_path(ped)):
9461                    log.debug("Pedigree is file")
9462                    with open(full_path(ped)) as ped:
9463                        ped = yaml.safe_load(ped)
9464
9465                # Pedigree is a string
9466                elif isinstance(ped, str):
9467                    log.debug("Pedigree is str")
9468                    try:
9469                        ped = json.loads(ped)
9470                        log.debug("Pedigree is json str")
9471                    except ValueError as e:
9472                        ped_samples = ped.split(",")
9473                        ped = {}
9474                        for ped_sample in ped_samples:
9475                            ped[ped_sample] = ped_sample
9476
9477                # Pedigree is a dict
9478                elif isinstance(ped, dict):
9479                    log.debug("Pedigree is dict")
9480
9481                # Pedigree is not well formatted
9482                else:
9483                    msg_error = "Pedigree not well formatted"
9484                    log.error(msg_error)
9485                    raise ValueError(msg_error)
9486
9487                # Construct list
9488                ped_samples = list(ped.values())
9489
9490            else:
9491                log.debug("Pedigree not defined. Take all samples")
9492                ped_samples = self.get_header_sample_list()
9493                ped = {}
9494                for ped_sample in ped_samples:
9495                    ped[ped_sample] = ped_sample
9496
9497            # Check pedigree
9498            if not ped or len(ped) == 0:
9499                msg_error = f"Error in pedigree: samples {ped_samples}"
9500                log.error(msg_error)
9501                raise ValueError(msg_error)
9502
9503            # Log
9504            log.info(
9505                "Calculation 'BARCODEFAMILY' - Samples: "
9506                + ", ".join([f"{member}='{ped[member]}'" for member in ped])
9507            )
9508            log.debug(f"ped_samples={ped_samples}")
9509
9510            # Field
9511            barcode_infos = prefix + tag
9512
9513            # Variants table
9514            table_variants = self.get_table_variants()
9515
9516            # Header
9517            vcf_reader = self.get_header()
9518
9519            # Create variant id
9520            variant_id_column = self.get_variant_id_column()
9521            added_columns = [variant_id_column]
9522
9523            # variant_id, FORMAT and samples
9524            samples_fields = f" {variant_id_column}, FORMAT , " + " , ".join(
9525                [f""" "{sample}" """ for sample in ped_samples]
9526            )
9527
9528            # Create dataframe
9529            dataframe_barcode = self.get_query_to_df(
9530                f""" SELECT {samples_fields} FROM {table_variants} """
9531            )
9532
9533            # Create barcode column
9534            dataframe_barcode[barcode_infos] = dataframe_barcode.apply(
9535                lambda row: barcode(row, samples=ped_samples), axis=1
9536            )
9537
9538            # Add barcode family to header
9539            # Add vaf_normalization to header
9540            vcf_reader.formats[tag] = vcf.parser._Format(
9541                id=tag,
9542                num=".",
9543                type="String",
9544                desc=vcf_infos_tags.get(tag, "barcode family calculation"),
9545                type_code=self.code_type_map.get("String"),
9546            )
9547            vcf_reader.formats[f"{tag}S"] = vcf.parser._Format(
9548                id=f"{tag}S",
9549                num=".",
9550                type="String",
9551                desc=vcf_infos_tags.get(f"{tag}S", "barcode family samples"),
9552                type_code=self.code_type_map.get("String"),
9553            )
9554
9555            # Update
9556            # for sample in ped_samples:
9557            sql_update_set = []
9558            for sample in self.get_header_sample_list() + ["FORMAT"]:
9559                if sample in ped_samples:
9560                    value = f'dataframe_barcode."{barcode_infos}"'
9561                    value_samples = (
9562                        "'"
9563                        + ",".join([f""" "{sample}" """ for sample in ped_samples])
9564                        + "'"
9565                    )
9566                    ped_samples
9567                elif sample == "FORMAT":
9568                    value = f"'{tag}'"
9569                    value_samples = f"'{tag}S'"
9570                else:
9571                    value = "'.'"
9572                    value_samples = "'.'"
9573                format_regex = r"[a-zA-Z0-9\s]"
9574                sql_update_set.append(
9575                    f"""
9576                        "{sample}" = 
9577                        concat(
9578                            CASE
9579                                WHEN {table_variants}."{sample}" = './.'
9580                                THEN concat('./.',regexp_replace(regexp_replace({table_variants}.FORMAT, '{format_regex}', '', 'g'), ':', ':.', 'g'))
9581                                ELSE {table_variants}."{sample}"
9582                            END,
9583                            ':',
9584                            {value},
9585                            ':',
9586                            {value_samples}
9587                        )
9588                    """
9589                )
9590
9591            sql_update_set_join = ", ".join(sql_update_set)
9592            sql_update = f"""
9593                UPDATE {table_variants}
9594                SET {sql_update_set_join}
9595                FROM dataframe_barcode
9596                WHERE {table_variants}."{variant_id_column}" = dataframe_barcode."{variant_id_column}"
9597            """
9598            self.conn.execute(sql_update)
9599
9600            # Remove added columns
9601            for added_column in added_columns:
9602                self.drop_column(column=added_column)
9603
9604            # Delete dataframe
9605            del dataframe_barcode
9606            gc.collect()

The calculation_barcode_family function calculates barcode values for variants in a VCF file and updates the INFO field in the file with the calculated barcode values.

Parameters

tag: The tag parameter in the calculation_barcode_family function is used to specify the barcode tag that will be added to the VCF file during the calculation process. If no value is provided for the tag parameter, the default value used is "BCF", defaults to BCF

def calculation_trio(self) -> None: View Source

9608    def calculation_trio(self) -> None:
9609        """
9610        The `calculation_trio` function performs trio calculations on a VCF file by adding trio
9611        information to the INFO field of each variant.
9612        """
9613
9614        # if FORMAT and samples
9615        if (
9616            "FORMAT" in self.get_header_columns_as_list()
9617            and self.get_header_sample_list()
9618        ):
9619
9620            # trio annotation field
9621            trio_tag = "trio"
9622
9623            # VCF infos tags
9624            vcf_infos_tags = {
9625                "trio": "trio calculation",
9626            }
9627
9628            # Param
9629            param = self.get_param()
9630
9631            # Prefix
9632            prefix = self.get_explode_infos_prefix()
9633
9634            # Trio param
9635            trio_ped = (
9636                param.get("calculation", {})
9637                .get("calculations", {})
9638                .get("TRIO", {})
9639                .get("trio_pedigree", None)
9640            )
9641
9642            # Load trio
9643            if trio_ped:
9644
9645                # Trio pedigree is a file
9646                if isinstance(trio_ped, str) and os.path.exists(full_path(trio_ped)):
9647                    log.debug("TRIO pedigree is file")
9648                    with open(full_path(trio_ped)) as trio_ped:
9649                        trio_ped = yaml.safe_load(trio_ped)
9650
9651                # Trio pedigree is a string
9652                elif isinstance(trio_ped, str):
9653                    log.debug("TRIO pedigree is str")
9654                    try:
9655                        trio_ped = json.loads(trio_ped)
9656                        log.debug("TRIO pedigree is json str")
9657                    except ValueError as e:
9658                        trio_samples = trio_ped.split(",")
9659                        if len(trio_samples) == 3:
9660                            trio_ped = {
9661                                "father": trio_samples[0],
9662                                "mother": trio_samples[1],
9663                                "child": trio_samples[2],
9664                            }
9665                            log.debug("TRIO pedigree is list str")
9666                        else:
9667                            msg_error = "TRIO pedigree not well formatted"
9668                            log.error(msg_error)
9669                            raise ValueError(msg_error)
9670
9671                # Trio pedigree is a dict
9672                elif isinstance(trio_ped, dict):
9673                    log.debug("TRIO pedigree is dict")
9674
9675                # Trio pedigree is not well formatted
9676                else:
9677                    msg_error = "TRIO pedigree not well formatted"
9678                    log.error(msg_error)
9679                    raise ValueError(msg_error)
9680
9681                # Construct trio list
9682                trio_samples = [
9683                    trio_ped.get("father", ""),
9684                    trio_ped.get("mother", ""),
9685                    trio_ped.get("child", ""),
9686                ]
9687
9688            else:
9689                log.debug("TRIO pedigree not defined. Take the first 3 samples")
9690                samples_list = self.get_header_sample_list()
9691                if len(samples_list) >= 3:
9692                    trio_samples = self.get_header_sample_list()[0:3]
9693                    trio_ped = {
9694                        "father": trio_samples[0],
9695                        "mother": trio_samples[1],
9696                        "child": trio_samples[2],
9697                    }
9698                else:
9699                    msg_error = f"Error in TRIO pedigree: only {len(samples_list)} samples {samples_list}"
9700                    log.error(msg_error)
9701                    raise ValueError(msg_error)
9702
9703            # Check trio pedigree
9704            if not trio_ped or len(trio_ped) != 3:
9705                msg_error = f"Error in TRIO pedigree: {trio_ped}"
9706                log.error(msg_error)
9707                raise ValueError(msg_error)
9708
9709            # Log
9710            log.info(
9711                f"Calculation 'TRIO' - Samples: "
9712                + ", ".join([f"{member}='{trio_ped[member]}'" for member in trio_ped])
9713            )
9714
9715            # Field
9716            trio_infos = prefix + trio_tag
9717
9718            # Variants table
9719            table_variants = self.get_table_variants()
9720
9721            # Header
9722            vcf_reader = self.get_header()
9723
9724            # Create variant id
9725            variant_id_column = self.get_variant_id_column()
9726            added_columns = [variant_id_column]
9727
9728            # variant_id, FORMAT and samples
9729            samples_fields = f" {variant_id_column}, FORMAT , " + " , ".join(
9730                [f""" "{sample}" """ for sample in self.get_header_sample_list()]
9731            )
9732
9733            # Create dataframe
9734            dataframe_trio = self.get_query_to_df(
9735                f""" SELECT {samples_fields} FROM {table_variants} """
9736            )
9737
9738            # Create trio column
9739            dataframe_trio[trio_infos] = dataframe_trio.apply(
9740                lambda row: trio(row, samples=trio_samples), axis=1
9741            )
9742
9743            # Add trio to header
9744            vcf_reader.infos[trio_tag] = vcf.parser._Info(
9745                trio_tag,
9746                ".",
9747                "String",
9748                vcf_infos_tags.get(trio_tag, "snpEff hgvs annotations"),
9749                "howard calculation",
9750                "0",
9751                self.code_type_map.get("String"),
9752            )
9753
9754            # Update
9755            sql_update = f"""
9756                UPDATE {table_variants}
9757                SET "INFO" = 
9758                    concat(
9759                        CASE
9760                            WHEN "INFO" IS NULL OR "INFO" IN ('','.')
9761                            THEN ''
9762                            ELSE concat("INFO", ';')
9763                        END,
9764                        CASE
9765                            WHEN dataframe_trio."{trio_infos}" NOT IN ('','.')
9766                             AND dataframe_trio."{trio_infos}" NOT NULL
9767                            THEN concat(
9768                                    '{trio_tag}=',
9769                                    dataframe_trio."{trio_infos}"
9770                                )
9771                            ELSE ''
9772                        END
9773                    )
9774                FROM dataframe_trio
9775                WHERE {table_variants}."{variant_id_column}" = dataframe_trio."{variant_id_column}"
9776            """
9777            self.conn.execute(sql_update)
9778
9779            # Remove added columns
9780            for added_column in added_columns:
9781                self.drop_column(column=added_column)
9782
9783            # Delete dataframe
9784            del dataframe_trio
9785            gc.collect()

The calculation_trio function performs trio calculations on a VCF file by adding trio information to the INFO field of each variant.

def calculation_vaf_normalization(self) -> None: View Source

9787    def calculation_vaf_normalization(self) -> None:
9788        """
9789        The `calculation_vaf_normalization` function calculates the VAF (Variant Allele Frequency)
9790        normalization for each sample in a VCF file and updates the FORMAT and INFO fields accordingly.
9791        :return: The function does not return anything.
9792        """
9793
9794        # if FORMAT and samples
9795        if (
9796            "FORMAT" in self.get_header_columns_as_list()
9797            and self.get_header_sample_list()
9798        ):
9799
9800            # vaf_normalization annotation field
9801            vaf_normalization_tag = "VAF"
9802
9803            # VCF infos tags
9804            vcf_infos_tags = {
9805                "VAF": "VAF Variant Frequency",
9806            }
9807
9808            # Prefix
9809            prefix = self.get_explode_infos_prefix()
9810
9811            # Variants table
9812            table_variants = self.get_table_variants()
9813
9814            # Header
9815            vcf_reader = self.get_header()
9816
9817            # Do not calculate if VAF already exists
9818            if "VAF" in vcf_reader.formats:
9819                log.debug("VAF already on genotypes")
9820                return
9821
9822            # Create variant id
9823            variant_id_column = self.get_variant_id_column()
9824            added_columns = [variant_id_column]
9825
9826            # variant_id, FORMAT and samples
9827            samples_fields = f" {variant_id_column}, FORMAT , " + " , ".join(
9828                f""" "{sample}" """ for sample in self.get_header_sample_list()
9829            )
9830
9831            # Create dataframe
9832            query = f""" SELECT {variant_id_column}, FORMAT, {samples_fields} FROM {table_variants} """
9833            log.debug(f"query={query}")
9834            dataframe_vaf_normalization = self.get_query_to_df(query=query)
9835
9836            vaf_normalization_set = []
9837
9838            # for each sample vaf_normalization
9839            for sample in self.get_header_sample_list():
9840                dataframe_vaf_normalization[sample] = dataframe_vaf_normalization.apply(
9841                    lambda row: vaf_normalization(row, sample=sample), axis=1
9842                )
9843                vaf_normalization_set.append(
9844                    f""" "{sample}" = dataframe_vaf_normalization."{sample}" """
9845                )
9846
9847            # Add VAF to FORMAT
9848            dataframe_vaf_normalization["FORMAT"] = dataframe_vaf_normalization[
9849                "FORMAT"
9850            ].apply(lambda x: str(x) + ":VAF")
9851            vaf_normalization_set.append(
9852                f""" "FORMAT" = dataframe_vaf_normalization."FORMAT" """
9853            )
9854
9855            # Add vaf_normalization to header
9856            vcf_reader.formats[vaf_normalization_tag] = vcf.parser._Format(
9857                id=vaf_normalization_tag,
9858                num="1",
9859                type="Float",
9860                desc=vcf_infos_tags.get(vaf_normalization_tag, "VAF Variant Frequency"),
9861                type_code=self.code_type_map.get("Float"),
9862            )
9863
9864            # Create fields to add in INFO
9865            sql_vaf_normalization_set = " , ".join(vaf_normalization_set)
9866
9867            # Update
9868            sql_update = f"""
9869                UPDATE {table_variants}
9870                SET {sql_vaf_normalization_set}
9871                FROM dataframe_vaf_normalization
9872                WHERE variants."{variant_id_column}" = dataframe_vaf_normalization."{variant_id_column}"
9873
9874            """
9875            self.conn.execute(sql_update)
9876
9877            # Remove added columns
9878            for added_column in added_columns:
9879                self.drop_column(column=added_column)
9880
9881            # Delete dataframe
9882            del dataframe_vaf_normalization
9883            gc.collect()

The calculation_vaf_normalization function calculates the VAF (Variant Allele Frequency) normalization for each sample in a VCF file and updates the FORMAT and INFO fields accordingly.

Returns

The function does not return anything.

def calculation_genotype_stats(self, info: str = 'VAF') -> None: View Source

 9885    def calculation_genotype_stats(self, info: str = "VAF") -> None:
 9886        """
 9887        The `calculation_genotype_stats` function calculates genotype statistics for a given information
 9888        field in a VCF file and updates the INFO column of the variants table with the calculated
 9889        statistics.
 9890
 9891        :param info: The `info` parameter is a string that represents the type of information for which
 9892        genotype statistics are calculated. It is used to generate various VCF info tags for the
 9893        statistics, such as the number of occurrences, the list of values, the minimum value, the
 9894        maximum value, the mean, the median, defaults to VAF
 9895        :type info: str (optional)
 9896        """
 9897
 9898        # if FORMAT and samples
 9899        if (
 9900            "FORMAT" in self.get_header_columns_as_list()
 9901            and self.get_header_sample_list()
 9902        ):
 9903
 9904            # vaf_stats annotation field
 9905            vaf_stats_tag = info + "_stats"
 9906
 9907            # VCF infos tags
 9908            vcf_infos_tags = {
 9909                info + "_stats_nb": f"genotype {info} Statistics - number of {info}",
 9910                info + "_stats_list": f"genotype {info} Statistics - list of {info}",
 9911                info + "_stats_min": f"genotype {info} Statistics - min {info}",
 9912                info + "_stats_max": f"genotype {info} Statistics - max {info}",
 9913                info + "_stats_mean": f"genotype {info} Statistics - mean {info}",
 9914                info + "_stats_mediane": f"genotype {info} Statistics - mediane {info}",
 9915                info
 9916                + "_stats_stdev": f"genotype {info} Statistics - standard deviation {info}",
 9917            }
 9918
 9919            # Prefix
 9920            prefix = self.get_explode_infos_prefix()
 9921
 9922            # Field
 9923            vaf_stats_infos = prefix + vaf_stats_tag
 9924
 9925            # Variants table
 9926            table_variants = self.get_table_variants()
 9927
 9928            # Header
 9929            vcf_reader = self.get_header()
 9930
 9931            # Create variant id
 9932            variant_id_column = self.get_variant_id_column()
 9933            added_columns = [variant_id_column]
 9934
 9935            # variant_id, FORMAT and samples
 9936            samples_fields = f" {variant_id_column}, FORMAT , " + " , ".join(
 9937                [f""" "{sample}" """ for sample in self.get_header_sample_list()]
 9938            )
 9939
 9940            # Create dataframe
 9941            dataframe_vaf_stats = self.get_query_to_df(
 9942                f""" SELECT {samples_fields} FROM {table_variants} """
 9943            )
 9944
 9945            # Create vaf_stats column
 9946            dataframe_vaf_stats[vaf_stats_infos] = dataframe_vaf_stats.apply(
 9947                lambda row: genotype_stats(
 9948                    row, samples=self.get_header_sample_list(), info=info
 9949                ),
 9950                axis=1,
 9951            )
 9952
 9953            # List of vcf tags
 9954            sql_vaf_stats_fields = []
 9955
 9956            # Check all VAF stats infos
 9957            for stat in vcf_infos_tags:
 9958
 9959                # Extract stats
 9960                dataframe_vaf_stats[stat] = dataframe_vaf_stats[vaf_stats_infos].apply(
 9961                    lambda x: dict(x).get(stat, "")
 9962                )
 9963
 9964                # Add snpeff_hgvs to header
 9965                vcf_reader.infos[stat] = vcf.parser._Info(
 9966                    stat,
 9967                    ".",
 9968                    "String",
 9969                    vcf_infos_tags.get(stat, "genotype statistics"),
 9970                    "howard calculation",
 9971                    "0",
 9972                    self.code_type_map.get("String"),
 9973                )
 9974
 9975                if len(sql_vaf_stats_fields):
 9976                    sep = ";"
 9977                else:
 9978                    sep = ""
 9979
 9980                # Create fields to add in INFO
 9981                sql_vaf_stats_fields.append(
 9982                    f"""
 9983                        CASE
 9984                            WHEN dataframe_vaf_stats."{stat}" NOT NULL
 9985                            THEN concat(
 9986                                    '{sep}{stat}=',
 9987                                    dataframe_vaf_stats."{stat}"
 9988                                )
 9989                            ELSE ''
 9990                        END
 9991                    """
 9992                )
 9993
 9994            # SQL set for update
 9995            sql_vaf_stats_fields_set = ",  ".join(sql_vaf_stats_fields)
 9996
 9997            # Update
 9998            sql_update = f"""
 9999                UPDATE {table_variants}
10000                SET "INFO" = 
10001                    concat(
10002                        CASE
10003                            WHEN "INFO" IS NULL OR "INFO" IN ('','.')
10004                            THEN ''
10005                            ELSE concat("INFO", ';')
10006                        END,
10007                        {sql_vaf_stats_fields_set}
10008                    )
10009                FROM dataframe_vaf_stats
10010                WHERE {table_variants}."{variant_id_column}" = dataframe_vaf_stats."{variant_id_column}"
10011
10012            """
10013            self.conn.execute(sql_update)
10014
10015            # Remove added columns
10016            for added_column in added_columns:
10017                self.drop_column(column=added_column)
10018
10019            # Delete dataframe
10020            del dataframe_vaf_stats
10021            gc.collect()

The calculation_genotype_stats function calculates genotype statistics for a given information field in a VCF file and updates the INFO column of the variants table with the calculated statistics.

Parameters

info: The info parameter is a string that represents the type of information for which genotype statistics are calculated. It is used to generate various VCF info tags for the statistics, such as the number of occurrences, the list of values, the minimum value, the maximum value, the mean, the median, defaults to VAF

def calculation_transcripts_annotation(self, info_json: str = None, info_format: str = None) -> None: View Source

10023    def calculation_transcripts_annotation(
10024        self, info_json: str = None, info_format: str = None
10025    ) -> None:
10026        """
10027        The `calculation_transcripts_annotation` function creates a transcripts table and adds an info
10028        field to it if transcripts are available.
10029
10030        :param info_json: The `info_json` parameter in the `calculation_transcripts_annotation` method
10031        is a string parameter that represents the information field to be used in the transcripts JSON.
10032        It is used to specify the JSON format for the transcripts information. If no value is provided
10033        when calling the method, it defaults to "
10034        :type info_json: str
10035        :param info_format: The `info_format` parameter in the `calculation_transcripts_annotation`
10036        method is a string parameter that specifies the format of the information field to be used in
10037        the transcripts JSON. It is used to define the format of the information field
10038        :type info_format: str
10039        """
10040
10041        # Create transcripts table
10042        transcripts_table = self.create_transcript_view()
10043
10044        # Add info field
10045        if transcripts_table:
10046            self.transcript_view_to_variants(
10047                transcripts_table=transcripts_table,
10048                transcripts_info_field_json=info_json,
10049                transcripts_info_field_format=info_format,
10050            )
10051        else:
10052            log.info("No Transcripts to process. Check param.json file configuration")

The calculation_transcripts_annotation function creates a transcripts table and adds an info field to it if transcripts are available.

Parameters

info_json: The info_json parameter in the calculation_transcripts_annotation method is a string parameter that represents the information field to be used in the transcripts JSON. It is used to specify the JSON format for the transcripts information. If no value is provided when calling the method, it defaults to "
info_format: The info_format parameter in the calculation_transcripts_annotation method is a string parameter that specifies the format of the information field to be used in the transcripts JSON. It is used to define the format of the information field

def calculation_transcripts_prioritization(self) -> None: View Source

10054    def calculation_transcripts_prioritization(self) -> None:
10055        """
10056        The function `calculation_transcripts_prioritization` creates a transcripts table and
10057        prioritizes transcripts based on certain criteria.
10058        """
10059
10060        # Create transcripts table
10061        transcripts_table = self.create_transcript_view()
10062
10063        # Add info field
10064        if transcripts_table:
10065            self.transcripts_prioritization(transcripts_table=transcripts_table)
10066        else:
10067            log.info("No Transcripts to process. Check param.json file configuration")

The function calculation_transcripts_prioritization creates a transcripts table and prioritizes transcripts based on certain criteria.

def calculation_transcripts_export(self) -> None: View Source

10069    def calculation_transcripts_export(self) -> None:
10070        """ """
10071
10072        # Create transcripts table
10073        transcripts_table = self.create_transcript_view()
10074
10075        # Add info field
10076        if transcripts_table:
10077            self.transcripts_export(transcripts_table=transcripts_table)
10078        else:
10079            log.info("No Transcripts to process. Check param.json file configuration")

def transcripts_export(self, transcripts_table: str = None, param: dict = {}) -> bool: View Source

10085    def transcripts_export(
10086        self, transcripts_table: str = None, param: dict = {}
10087    ) -> bool:
10088        """ """
10089
10090        log.debug("Start transcripts export...")
10091
10092        # Param
10093        if not param:
10094            param = self.get_param()
10095
10096        # Param export
10097        param_transcript_export = param.get("transcripts", {}).get("export", {})
10098
10099        # Output file
10100        transcripts_export_output = param_transcript_export.get("output", None)
10101
10102        if not param_transcript_export or not transcripts_export_output:
10103            log.warning(f"No transcriipts export parameters defined!")
10104            return False
10105
10106        # List of transcripts annotations
10107        query_describe = f"""
10108            SELECT column_name
10109            FROM (
10110                    DESCRIBE SELECT * FROM {transcripts_table}
10111                )
10112            WHERE column_name NOT IN ('#CHROM', 'POS', 'REF', 'ALT', 'INFO')
10113        """
10114        transcripts_annotations_list = list(
10115            self.get_query_to_df(query=query_describe)["column_name"]
10116        )
10117
10118        # Create transcripts table for export
10119        transcripts_table_export = f"{transcripts_table}_export_" + "".join(
10120            random.choices(string.ascii_uppercase + string.digits, k=10)
10121        )
10122        query_create_transcripts_table_export = f"""
10123            CREATE TABLE {transcripts_table_export} AS (SELECT "#CHROM", "POS", "REF", "ALT", '' AS 'INFO', {', '.join(transcripts_annotations_list)} FROM {transcripts_table})
10124        """
10125        self.execute_query(query=query_create_transcripts_table_export)
10126
10127        # Output file format
10128        transcripts_export_output_format = get_file_format(
10129            filename=transcripts_export_output
10130        )
10131
10132        # Format VCF - construct INFO
10133        if transcripts_export_output_format in ["vcf"]:
10134
10135            # Construct query update INFO and header
10136            query_update_info = []
10137            for field in transcripts_annotations_list:
10138
10139                # If field not in header
10140                if field not in self.get_header_infos_list():
10141
10142                    # Add PZ Transcript in header
10143                    self.get_header().infos[field] = vcf.parser._Info(
10144                        field,
10145                        ".",
10146                        "String",
10147                        f"Annotation '{field}' from transcript view",
10148                        "unknown",
10149                        "unknown",
10150                        0,
10151                    )
10152
10153                # Add field as INFO/tag
10154                query_update_info.append(
10155                    f"""
10156                        CASE
10157                            WHEN "{field}" IS NOT NULL
10158                            THEN concat('{field}=', "{field}", ';')    
10159                            ELSE ''     
10160                        END
10161                        """
10162                )
10163
10164            # Query param
10165            query_update_info_value = (
10166                f""" concat('',  {", ".join(query_update_info)}) """
10167            )
10168            query_export_columns = f""" "#CHROM", "POS", '.' AS 'ID', "REF", "ALT", '.' AS 'QUAL', '.' AS 'FILTER', "INFO" """
10169
10170        else:
10171
10172            # Query param
10173            query_update_info_value = f""" NULL """
10174            query_export_columns = f""" "#CHROM", "POS", "REF", "ALT", {', '.join(transcripts_annotations_list)} """
10175
10176        # Update query INFO column
10177        query_update = f"""
10178            UPDATE {transcripts_table_export}
10179            SET INFO = {query_update_info_value}
10180
10181        """
10182        self.execute_query(query=query_update)
10183
10184        # Export
10185        self.export_output(
10186            output_file=transcripts_export_output,
10187            query=f""" SELECT {query_export_columns} FROM {transcripts_table_export} """,
10188        )
10189
10190        # Drop transcripts export table
10191        query_drop_transcripts_table_export = f"""
10192            DROP TABLE {transcripts_table_export}
10193        """
10194        self.execute_query(query=query_drop_transcripts_table_export)

def transcripts_prioritization(self, transcripts_table: str = None, param: dict = {}) -> bool: View Source

10196    def transcripts_prioritization(
10197        self, transcripts_table: str = None, param: dict = {}
10198    ) -> bool:
10199        """
10200        The `transcripts_prioritization` function prioritizes transcripts based on certain parameters
10201        and updates the variants table with the prioritized information.
10202
10203        :param transcripts_table: The `transcripts_table` parameter is a string that specifies the name
10204        of the table containing transcripts data. If no value is provided, it defaults to "transcripts".
10205        This parameter is used to identify the table where the transcripts data is stored for the
10206        prioritization process
10207        :type transcripts_table: str
10208        :param param: The `param` parameter in the `transcripts_prioritization` method is a dictionary
10209        that contains various configuration settings for the prioritization process of transcripts. It
10210        is used to customize the behavior of the prioritization algorithm and includes settings such as
10211        the prefix for prioritization fields, default profiles, and other
10212        :type param: dict
10213        :return: The function `transcripts_prioritization` returns a boolean value `True` if the
10214        transcripts prioritization process is successfully completed, and `False` if there are any
10215        issues or if no profile is defined for transcripts prioritization.
10216        """
10217
10218        log.debug("Start transcripts prioritization...")
10219
10220        # Param
10221        if not param:
10222            param = self.get_param()
10223
10224        # Variants table
10225        table_variants = self.get_table_variants()
10226
10227        # Transcripts table
10228        if transcripts_table is None:
10229            transcripts_table = self.create_transcript_view(
10230                transcripts_table="transcripts", param=param
10231            )
10232        if transcripts_table is None:
10233            msg_err = "No Transcripts table availalble"
10234            log.error(msg_err)
10235            raise ValueError(msg_err)
10236        log.debug(f"transcripts_table={transcripts_table}")
10237
10238        # Get transcripts columns
10239        columns_as_list_query = f"""
10240            DESCRIBE {transcripts_table}
10241        """
10242        columns_as_list = list(
10243            self.get_query_to_df(columns_as_list_query)["column_name"]
10244        )
10245
10246        # Create INFO if not exists
10247        if "INFO" not in columns_as_list:
10248            query_add_info = f"""
10249                ALTER TABLE {transcripts_table} ADD COLUMN INFO STRING DEFAULT '';
10250            """
10251            self.execute_query(query_add_info)
10252
10253        # Prioritization param and Force only PZ Score and Flag
10254        pz_param = param.get("transcripts", {}).get("prioritization", {})
10255
10256        # PZ profile by default
10257        pz_profile_default = (
10258            param.get("transcripts", {}).get("prioritization", {}).get("profiles", None)
10259        )
10260
10261        # Exit if no profile
10262        if pz_profile_default is None:
10263            log.warning("No profile defined for transcripts prioritization")
10264            return False
10265
10266        # PZ fields
10267        pz_param_pzfields = {}
10268
10269        # PZ field transcripts
10270        pz_fields_transcripts = pz_param.get("pzprefix", "PTZ") + "Transcript"
10271
10272        # Add PZ Transcript in header
10273        self.get_header().infos[pz_fields_transcripts] = vcf.parser._Info(
10274            pz_fields_transcripts,
10275            ".",
10276            "String",
10277            f"Transcript selected from prioritization process, profile {pz_profile_default}",
10278            "unknown",
10279            "unknown",
10280            code_type_map["String"],
10281        )
10282
10283        # Mandatory fields
10284        pz_mandatory_fields_list = [
10285            "Score",
10286            "Flag",
10287            "Tags",
10288            "Comment",
10289            "Infos",
10290            "Class",
10291        ]
10292        pz_mandatory_fields = []
10293        for pz_mandatory_field in pz_mandatory_fields_list:
10294            pz_mandatory_fields.append(
10295                pz_param.get("pzprefix", "PTZ") + pz_mandatory_field
10296            )
10297
10298        # PZ fields in param
10299        for pz_field in pz_param.get("pzfields", []):
10300            if pz_field in pz_mandatory_fields_list:
10301                pz_param_pzfields[pz_param.get("pzprefix", "PTZ") + pz_field] = (
10302                    pz_param.get("pzprefix", "PTZ") + pz_field
10303                )
10304            else:
10305                pz_field_new = pz_param.get("pzprefix", "PTZ") + pz_field
10306                pz_param_pzfields[pz_field] = pz_field_new
10307
10308                # Add PZ Transcript in header
10309                self.get_header().infos[pz_field_new] = vcf.parser._Info(
10310                    pz_field_new,
10311                    ".",
10312                    "String",
10313                    f"Annotation '{pz_field}' from transcript selected from prioritization process, profile {pz_profile_default}",
10314                    "unknown",
10315                    "unknown",
10316                    code_type_map["String"],
10317                )
10318
10319        # PZ fields param
10320        pz_param["pzfields"] = pz_mandatory_fields
10321
10322        # Prioritization
10323        prioritization_result = self.prioritization(
10324            table=transcripts_table,
10325            pz_param=param.get("transcripts", {}).get("prioritization", {}),
10326        )
10327        if not prioritization_result:
10328            log.warning("Transcripts prioritization not processed")
10329            return False
10330
10331        # PZ fields sql query
10332        query_update_select_list = []
10333        query_update_concat_list = []
10334        query_update_order_list = []
10335        for pz_param_pzfield in set(
10336            list(pz_param_pzfields.keys()) + pz_mandatory_fields
10337        ):
10338            query_update_select_list.append(f" {pz_param_pzfield}, ")
10339
10340        for pz_param_pzfield in pz_param_pzfields:
10341            query_update_concat_list.append(
10342                f"""
10343                    , CASE 
10344                        WHEN {pz_param_pzfield} IS NOT NULL
10345                        THEN concat(';{pz_param_pzfields.get(pz_param_pzfield)}=', {pz_param_pzfield})
10346                        ELSE ''
10347                    END
10348                """
10349            )
10350
10351        # Order by
10352        pz_orders = (
10353            param.get("transcripts", {})
10354            .get("prioritization", {})
10355            .get("prioritization_transcripts_order", {})
10356        )
10357        if not pz_orders:
10358            pz_orders = {
10359                pz_param.get("pzprefix", "PTZ") + "Flag": "DESC",
10360                pz_param.get("pzprefix", "PTZ") + "Score": "DESC",
10361            }
10362        for pz_order in pz_orders:
10363            query_update_order_list.append(
10364                f""" {pz_order} {pz_orders.get(pz_order, "DESC")} """
10365            )
10366
10367        # Fields to explode
10368        fields_to_explode = (
10369            list(pz_param_pzfields.keys())
10370            + pz_mandatory_fields
10371            + list(pz_orders.keys())
10372        )
10373        # Remove transcript column as a specific transcript column
10374        if "transcript" in fields_to_explode:
10375            fields_to_explode.remove("transcript")
10376
10377        # Fields intranscripts table
10378        query_transcripts_table = f"""
10379            DESCRIBE SELECT * FROM {transcripts_table}
10380        """
10381        query_transcripts_table = self.get_query_to_df(query=query_transcripts_table)
10382
10383        # Check fields to explode
10384        for field_to_explode in fields_to_explode:
10385            if field_to_explode not in self.get_header_infos_list() + list(
10386                query_transcripts_table.column_name
10387            ):
10388                msg_err = f"INFO/{field_to_explode} NOT IN header"
10389                log.error(msg_err)
10390                raise ValueError(msg_err)
10391
10392        # Explode fields to explode
10393        self.explode_infos(
10394            table=transcripts_table,
10395            fields=fields_to_explode,
10396        )
10397
10398        # Transcript preference file
10399        transcripts_preference_file = (
10400            param.get("transcripts", {})
10401            .get("prioritization", {})
10402            .get("prioritization_transcripts", {})
10403        )
10404        transcripts_preference_file = full_path(transcripts_preference_file)
10405
10406        # Transcript preference forced
10407        transcript_preference_force = (
10408            param.get("transcripts", {})
10409            .get("prioritization", {})
10410            .get("prioritization_transcripts_force", False)
10411        )
10412        # Transcript version forced
10413        transcript_version_force = (
10414            param.get("transcripts", {})
10415            .get("prioritization", {})
10416            .get("prioritization_transcripts_version_force", False)
10417        )
10418
10419        # Transcripts Ranking
10420        if transcripts_preference_file:
10421
10422            # Transcripts file to dataframe
10423            if os.path.exists(transcripts_preference_file):
10424                transcripts_preference_dataframe = transcripts_file_to_df(
10425                    transcripts_preference_file
10426                )
10427            else:
10428                log.error(
10429                    f"Transcript file '{transcripts_preference_file}' does NOT exist"
10430                )
10431                raise ValueError(
10432                    f"Transcript file '{transcripts_preference_file}' does NOT exist"
10433                )
10434
10435            # Order by depending to transcript preference forcing
10436            if transcript_preference_force:
10437                order_by = f""" transcripts_preference.transcripts_preference_order ASC, {" , ".join(query_update_order_list)} """
10438            else:
10439                order_by = f""" {" , ".join(query_update_order_list)}, transcripts_preference.transcripts_preference_order ASC """
10440
10441            # Transcript columns joined depend on version consideration
10442            if transcript_version_force:
10443                transcripts_version_join = f""" {transcripts_table}.transcript = transcripts_preference.transcripts_preference """
10444            else:
10445                transcripts_version_join = f""" split_part({transcripts_table}.transcript, '.', 1) = split_part(transcripts_preference.transcripts_preference, '.', 1) """
10446
10447            # Query ranking for update
10448            query_update_ranking = f"""
10449                SELECT
10450                    "#CHROM", POS, REF, ALT, {transcripts_table}.transcript, {" ".join(query_update_select_list)}
10451                    ROW_NUMBER() OVER (
10452                        PARTITION BY "#CHROM", POS, REF, ALT
10453                        ORDER BY {order_by}
10454                    ) AS rn
10455                FROM {transcripts_table}
10456                LEFT JOIN 
10457                    (
10458                        SELECT transcript AS 'transcripts_preference', row_number() OVER () AS transcripts_preference_order
10459                        FROM transcripts_preference_dataframe
10460                    ) AS transcripts_preference
10461                ON {transcripts_version_join}
10462            """
10463
10464        else:
10465
10466            # Query ranking for update
10467            query_update_ranking = f"""
10468                SELECT
10469                    "#CHROM", POS, REF, ALT, transcript, {" ".join(query_update_select_list)}
10470                    ROW_NUMBER() OVER (
10471                        PARTITION BY "#CHROM", POS, REF, ALT
10472                        ORDER BY {" , ".join(query_update_order_list)}
10473                    ) AS rn
10474                FROM {transcripts_table}
10475            """
10476
10477        # Export Transcripts prioritization infos to variants table
10478        query_update = f"""
10479            WITH RankedTranscripts AS (
10480                {query_update_ranking}
10481            )
10482            UPDATE {table_variants}
10483                SET
10484                INFO = CONCAT(CASE
10485                            WHEN "INFO" IS NULL OR "INFO" IN ('','.')
10486                            THEN ''
10487                            ELSE concat("INFO", ';')
10488                        END,
10489                        concat('{pz_fields_transcripts}=', transcript {" ".join(query_update_concat_list)})
10490                        )
10491            FROM
10492                RankedTranscripts
10493            WHERE
10494                rn = 1
10495                AND variants."#CHROM" = RankedTranscripts."#CHROM"
10496                AND variants."POS" = RankedTranscripts."POS"
10497                AND variants."REF" = RankedTranscripts."REF"
10498                AND variants."ALT" = RankedTranscripts."ALT"     
10499        """
10500
10501        # log.debug(f"query_update={query_update}")
10502        self.execute_query(query=query_update)
10503
10504        # Return
10505        return True

The transcripts_prioritization function prioritizes transcripts based on certain parameters and updates the variants table with the prioritized information.

Parameters

transcripts_table: The transcripts_table parameter is a string that specifies the name of the table containing transcripts data. If no value is provided, it defaults to "transcripts". This parameter is used to identify the table where the transcripts data is stored for the prioritization process
param: The param parameter in the transcripts_prioritization method is a dictionary that contains various configuration settings for the prioritization process of transcripts. It is used to customize the behavior of the prioritization algorithm and includes settings such as the prefix for prioritization fields, default profiles, and other

Returns

The function transcripts_prioritization returns a boolean value True if the transcripts prioritization process is successfully completed, and False if there are any issues or if no profile is defined for transcripts prioritization.

def create_transcript_view_from_columns_map( self, transcripts_table: str = 'transcripts', columns_maps: dict = {}, added_columns: list = [], temporary_tables: list = None, annotation_fields: list = None, column_rename: dict = {}, column_clean: bool = False, column_case: str = None) -> tuple[list, list, list]: View Source

10507    def create_transcript_view_from_columns_map(
10508        self,
10509        transcripts_table: str = "transcripts",
10510        columns_maps: dict = {},
10511        added_columns: list = [],
10512        temporary_tables: list = None,
10513        annotation_fields: list = None,
10514        column_rename: dict = {},
10515        column_clean: bool = False,
10516        column_case: str = None,
10517    ) -> tuple[list, list, list]:
10518        """
10519        The `create_transcript_view_from_columns_map` function generates a temporary table view based on
10520        specified columns mapping for transcripts data.
10521
10522        :param transcripts_table: The `transcripts_table` parameter is a string that specifies the name
10523        of the table where the transcripts data is stored or will be stored in the database. This table
10524        typically contains information about transcripts such as Ensembl transcript IDs, gene names,
10525        scores, predictions, etc. It defaults to "transcripts, defaults to transcripts
10526        :type transcripts_table: str (optional)
10527        :param columns_maps: The `columns_maps` parameter is a dictionary that contains information
10528        about how to map columns from a transcripts table to create a view. Each entry in the
10529        `columns_maps` list represents a mapping configuration for a specific set of columns. It
10530        typically includes details such as the main transcript column and additional information columns
10531        :type columns_maps: dict
10532        :param added_columns: The `added_columns` parameter in the
10533        `create_transcript_view_from_columns_map` function is a list that stores the additional columns
10534        that will be added to the view being created based on the columns map provided. These columns
10535        are generated by exploding the transcript information columns along with the main transcript
10536        column
10537        :type added_columns: list
10538        :param temporary_tables: The `temporary_tables` parameter in the
10539        `create_transcript_view_from_columns_map` function is a list that stores the names of temporary
10540        tables created during the process of creating a transcript view from a columns map. These
10541        temporary tables are used to store intermediate results or transformations before the final view
10542        is generated
10543        :type temporary_tables: list
10544        :param annotation_fields: The `annotation_fields` parameter in the
10545        `create_transcript_view_from_columns_map` function is a list that stores the fields that are
10546        used for annotation in the query view creation process. These fields are extracted from the
10547        `transcripts_column` and `transcripts_infos_columns` specified in the `columns
10548        :type annotation_fields: list
10549        :param column_rename: The `column_rename` parameter in the
10550        `create_transcript_view_from_columns_map` function is a dictionary that allows you to specify
10551        custom renaming for columns during the creation of the temporary table view. This parameter
10552        provides a mapping of original column names to the desired renamed column names. By using this
10553        parameter,
10554        :type column_rename: dict
10555        :param column_clean: The `column_clean` parameter in the
10556        `create_transcript_view_from_columns_map` function is a boolean flag that determines whether the
10557        column values should be cleaned or not. If set to `True`, the column values will be cleaned by
10558        removing any non-alphanumeric characters from them. This cleaning process ensures, defaults to
10559        False
10560        :type column_clean: bool (optional)
10561        :param column_case: The `column_case` parameter in the `create_transcript_view_from_columns_map`
10562        function is used to specify the case transformation to be applied to the columns during the view
10563        creation process. It allows you to control whether the column values should be converted to
10564        lowercase, uppercase, or remain unchanged
10565        :type column_case: str
10566        :return: The `create_transcript_view_from_columns_map` function returns a tuple containing three
10567        lists: `added_columns`, `temporary_tables`, and `annotation_fields`.
10568        """
10569
10570        log.debug("Start transcrpts view creation from columns map...")
10571
10572        # "from_columns_map": [
10573        #     {
10574        #         "transcripts_column": "Ensembl_transcriptid",
10575        #         "transcripts_infos_columns": [
10576        #             "genename",
10577        #             "Ensembl_geneid",
10578        #             "LIST_S2_score",
10579        #             "LIST_S2_pred",
10580        #         ],
10581        #     },
10582        #     {
10583        #         "transcripts_column": "Ensembl_transcriptid",
10584        #         "transcripts_infos_columns": [
10585        #             "genename",
10586        #             "VARITY_R_score",
10587        #             "Aloft_pred",
10588        #         ],
10589        #     },
10590        # ],
10591
10592        # Init
10593        if temporary_tables is None:
10594            temporary_tables = []
10595        if annotation_fields is None:
10596            annotation_fields = []
10597
10598        # Variants table
10599        table_variants = self.get_table_variants()
10600
10601        for columns_map in columns_maps:
10602
10603            # Log
10604            log.debug(f"columns_map={columns_map}")
10605
10606            # Transcript column
10607            transcripts_column = columns_map.get("transcripts_column", None)
10608
10609            # Transcripts infos columns
10610            transcripts_infos_columns = columns_map.get("transcripts_infos_columns", [])
10611
10612            # Transcripts infos columns rename
10613            column_rename = columns_map.get("column_rename", column_rename)
10614
10615            # Transcripts infos columns clean
10616            column_clean = columns_map.get("column_clean", column_clean)
10617
10618            # Transcripts infos columns case
10619            column_case = columns_map.get("column_case", column_case)
10620
10621            if transcripts_column is not None:
10622
10623                # Explode
10624                added_columns += self.explode_infos(
10625                    fields=[transcripts_column] + transcripts_infos_columns
10626                )
10627
10628                # View clauses
10629                clause_select_variants = []
10630                clause_select_tanscripts = []
10631                for field in [transcripts_column] + transcripts_infos_columns:
10632
10633                    # AS field
10634                    as_field = field
10635
10636                    # Rename
10637                    if column_rename:
10638                        as_field = column_rename.get(as_field, as_field)
10639
10640                    # Clean
10641                    if column_clean:
10642                        as_field = clean_annotation_field(as_field)
10643
10644                    # Case
10645                    if column_case:
10646                        if column_case.lower() in ["lower"]:
10647                            as_field = as_field.lower()
10648                        elif column_case.lower() in ["upper"]:
10649                            as_field = as_field.upper()
10650
10651                    # Clause select Variants
10652                    clause_select_variants.append(
10653                        f""" regexp_split_to_table("{field}", ',') AS '{field}' """
10654                    )
10655
10656                    if field in [transcripts_column]:
10657                        clause_select_tanscripts.append(
10658                            f""" regexp_split_to_table("{field}", ',') AS '{field}' """
10659                        )
10660                    else:
10661                        clause_select_tanscripts.append(
10662                            f""" regexp_split_to_table("{field}", ',') AS '{as_field}' """
10663                        )
10664                        annotation_fields.append(as_field)
10665
10666                # Query View
10667                query = f""" 
10668                    SELECT
10669                        "#CHROM", POS, REF, ALT, INFO,
10670                        "{transcripts_column}" AS 'transcript',
10671                        {", ".join(clause_select_tanscripts)}
10672                    FROM (
10673                        SELECT 
10674                            "#CHROM", POS, REF, ALT, INFO,
10675                            {", ".join(clause_select_variants)}
10676                        FROM {table_variants}
10677                        )
10678                    WHERE "{transcripts_column}" IS NOT NULL
10679                """
10680
10681                # Create temporary table
10682                temporary_table = transcripts_table + "".join(
10683                    random.choices(string.ascii_uppercase + string.digits, k=10)
10684                )
10685
10686                # # Temporary_tables
10687                # temporary_tables.append(temporary_table)
10688                # query_view = f"""
10689                #     CREATE TEMPORARY TABLE {temporary_table}
10690                #     AS ({query})
10691                # """
10692                # self.execute_query(query=query_view)
10693
10694                # Temporary_tables
10695                temporary_tables.append(temporary_table)
10696
10697                # List of unique #CHROM
10698                query_unique_chrom = f"""
10699                    SELECT DISTINCT "#CHROM"
10700                    FROM variants
10701                """
10702                unique_chroms = self.get_query_to_df(query=query_unique_chrom)
10703
10704                # Create table with structure but without data
10705                query_create_table = f"""
10706                    CREATE TABLE {temporary_table}
10707                    AS ({query} LIMIT 0)
10708                """
10709                self.execute_query(query=query_create_table)
10710
10711                # Process by #CHROM
10712                for chrom in unique_chroms["#CHROM"]:
10713
10714                    # Log
10715                    log.debug(f"Processing #CHROM={chrom}")
10716
10717                    # Select data by #CHROM
10718                    query_chunk = f"""
10719                        SELECT *
10720                        FROM ({query})
10721                        WHERE "#CHROM" = '{chrom}'
10722                    """
10723
10724                    # Insert data
10725                    query_insert_chunk = f"""
10726                        INSERT INTO {temporary_table}
10727                        {query_chunk}
10728                    """
10729                    self.execute_query(query=query_insert_chunk)
10730
10731        return added_columns, temporary_tables, annotation_fields

The create_transcript_view_from_columns_map function generates a temporary table view based on specified columns mapping for transcripts data.

Parameters

transcripts_table: The transcripts_table parameter is a string that specifies the name of the table where the transcripts data is stored or will be stored in the database. This table typically contains information about transcripts such as Ensembl transcript IDs, gene names, scores, predictions, etc. It defaults to "transcripts, defaults to transcripts
columns_maps: The columns_maps parameter is a dictionary that contains information about how to map columns from a transcripts table to create a view. Each entry in the columns_maps list represents a mapping configuration for a specific set of columns. It typically includes details such as the main transcript column and additional information columns
added_columns: The added_columns parameter in the create_transcript_view_from_columns_map function is a list that stores the additional columns that will be added to the view being created based on the columns map provided. These columns are generated by exploding the transcript information columns along with the main transcript column
temporary_tables: The temporary_tables parameter in the create_transcript_view_from_columns_map function is a list that stores the names of temporary tables created during the process of creating a transcript view from a columns map. These temporary tables are used to store intermediate results or transformations before the final view is generated
annotation_fields: The annotation_fields parameter in the create_transcript_view_from_columns_map function is a list that stores the fields that are used for annotation in the query view creation process. These fields are extracted from the transcripts_column and transcripts_infos_columns specified in the `columns
column_rename: The column_rename parameter in the create_transcript_view_from_columns_map function is a dictionary that allows you to specify custom renaming for columns during the creation of the temporary table view. This parameter provides a mapping of original column names to the desired renamed column names. By using this parameter,
column_clean: The column_clean parameter in the create_transcript_view_from_columns_map function is a boolean flag that determines whether the column values should be cleaned or not. If set to True, the column values will be cleaned by removing any non-alphanumeric characters from them. This cleaning process ensures, defaults to False
column_case: The column_case parameter in the create_transcript_view_from_columns_map function is used to specify the case transformation to be applied to the columns during the view creation process. It allows you to control whether the column values should be converted to lowercase, uppercase, or remain unchanged

Returns

The create_transcript_view_from_columns_map function returns a tuple containing three lists: added_columns, temporary_tables, and annotation_fields.

def create_transcript_view_from_column_format( self, transcripts_table: str = 'transcripts', column_formats: dict = {}, temporary_tables: list = None, annotation_fields: list = None, column_rename: dict = {}, column_clean: bool = False, column_case: str = None) -> tuple[list, list, list]: View Source

10733    def create_transcript_view_from_column_format(
10734        self,
10735        transcripts_table: str = "transcripts",
10736        column_formats: dict = {},
10737        temporary_tables: list = None,
10738        annotation_fields: list = None,
10739        column_rename: dict = {},
10740        column_clean: bool = False,
10741        column_case: str = None,
10742    ) -> tuple[list, list, list]:
10743        """
10744        The `create_transcript_view_from_column_format` function generates a transcript view based on
10745        specified column formats, adds additional columns and annotation fields, and returns the list of
10746        temporary tables and annotation fields.
10747
10748        :param transcripts_table: The `transcripts_table` parameter is a string that specifies the name
10749        of the table containing the transcripts data. This table will be used as the base table for
10750        creating the transcript view. The default value for this parameter is "transcripts", but you can
10751        provide a different table name if needed, defaults to transcripts
10752        :type transcripts_table: str (optional)
10753        :param column_formats: The `column_formats` parameter is a dictionary that contains information
10754        about the columns to be used for creating the transcript view. Each entry in the dictionary
10755        specifies the mapping between a transcripts column and a transcripts infos column. This
10756        parameter allows you to define how the columns from the transcripts table should be transformed
10757        or mapped
10758        :type column_formats: dict
10759        :param temporary_tables: The `temporary_tables` parameter in the
10760        `create_transcript_view_from_column_format` function is a list that stores the names of
10761        temporary views created during the process of creating a transcript view from a column format.
10762        These temporary views are used to manipulate and extract data before generating the final
10763        transcript view
10764        :type temporary_tables: list
10765        :param annotation_fields: The `annotation_fields` parameter in the
10766        `create_transcript_view_from_column_format` function is a list that stores the annotation fields
10767        that are extracted from the temporary views created during the process. These annotation fields
10768        are obtained by querying the temporary views and extracting the column names excluding specific
10769        columns like `#CH
10770        :type annotation_fields: list
10771        :param column_rename: The `column_rename` parameter in the
10772        `create_transcript_view_from_column_format` function is a dictionary that allows you to specify
10773        custom renaming of columns in the transcripts infos table. By providing a mapping of original
10774        column names to new column names in this dictionary, you can rename specific columns during the
10775        process
10776        :type column_rename: dict
10777        :param column_clean: The `column_clean` parameter in the
10778        `create_transcript_view_from_column_format` function is a boolean flag that determines whether
10779        the transcripts infos columns should undergo a cleaning process. If set to `True`, the columns
10780        will be cleaned during the creation of the transcript view based on the specified column format,
10781        defaults to False
10782        :type column_clean: bool (optional)
10783        :param column_case: The `column_case` parameter in the
10784        `create_transcript_view_from_column_format` function is used to specify the case transformation
10785        to be applied to the columns in the transcript view. It can be set to either "upper" or "lower"
10786        to convert the column names to uppercase or lowercase, respectively
10787        :type column_case: str
10788        :return: The `create_transcript_view_from_column_format` function returns two lists:
10789        `temporary_tables` and `annotation_fields`.
10790        """
10791
10792        log.debug("Start transcrpts view creation from column format...")
10793
10794        #  "from_column_format": [
10795        #     {
10796        #         "transcripts_column": "ANN",
10797        #         "transcripts_infos_column": "Feature_ID",
10798        #     }
10799        # ],
10800
10801        # Init
10802        if temporary_tables is None:
10803            temporary_tables = []
10804        if annotation_fields is None:
10805            annotation_fields = []
10806
10807        for column_format in column_formats:
10808
10809            # annotation field and transcript annotation field
10810            annotation_field = column_format.get("transcripts_column", "ANN")
10811            transcript_annotation = column_format.get(
10812                "transcripts_infos_column", "Feature_ID"
10813            )
10814
10815            # Transcripts infos columns rename
10816            column_rename = column_format.get("column_rename", column_rename)
10817
10818            # Transcripts infos columns clean
10819            column_clean = column_format.get("column_clean", column_clean)
10820
10821            # Transcripts infos columns case
10822            column_case = column_format.get("column_case", column_case)
10823
10824            # Temporary View name
10825            temporary_view_name = transcripts_table + "".join(
10826                random.choices(string.ascii_uppercase + string.digits, k=10)
10827            )
10828
10829            # Create temporary view name
10830            temporary_view_name = self.annotation_format_to_table(
10831                uniquify=True,
10832                annotation_field=annotation_field,
10833                view_name=temporary_view_name,
10834                annotation_id=transcript_annotation,
10835                column_rename=column_rename,
10836                column_clean=column_clean,
10837                column_case=column_case,
10838            )
10839
10840            # Annotation fields
10841            if temporary_view_name:
10842                query_annotation_fields = f"""
10843                    SELECT *
10844                    FROM (
10845                        DESCRIBE SELECT *
10846                        FROM {temporary_view_name}
10847                        )
10848                        WHERE column_name not in ('#CHROM', 'POS', 'REF', 'ALT')
10849                """
10850                df_annotation_fields = self.get_query_to_df(
10851                    query=query_annotation_fields
10852                )
10853
10854                # Add temporary view and annotation fields
10855                temporary_tables.append(temporary_view_name)
10856                annotation_fields += list(set(df_annotation_fields["column_name"]))
10857
10858        return temporary_tables, annotation_fields

The create_transcript_view_from_column_format function generates a transcript view based on specified column formats, adds additional columns and annotation fields, and returns the list of temporary tables and annotation fields.

Parameters

transcripts_table: The transcripts_table parameter is a string that specifies the name of the table containing the transcripts data. This table will be used as the base table for creating the transcript view. The default value for this parameter is "transcripts", but you can provide a different table name if needed, defaults to transcripts
column_formats: The column_formats parameter is a dictionary that contains information about the columns to be used for creating the transcript view. Each entry in the dictionary specifies the mapping between a transcripts column and a transcripts infos column. This parameter allows you to define how the columns from the transcripts table should be transformed or mapped
temporary_tables: The temporary_tables parameter in the create_transcript_view_from_column_format function is a list that stores the names of temporary views created during the process of creating a transcript view from a column format. These temporary views are used to manipulate and extract data before generating the final transcript view
annotation_fields: The annotation_fields parameter in the create_transcript_view_from_column_format function is a list that stores the annotation fields that are extracted from the temporary views created during the process. These annotation fields are obtained by querying the temporary views and extracting the column names excluding specific columns like `#CH
column_rename: The column_rename parameter in the create_transcript_view_from_column_format function is a dictionary that allows you to specify custom renaming of columns in the transcripts infos table. By providing a mapping of original column names to new column names in this dictionary, you can rename specific columns during the process
column_clean: The column_clean parameter in the create_transcript_view_from_column_format function is a boolean flag that determines whether the transcripts infos columns should undergo a cleaning process. If set to True, the columns will be cleaned during the creation of the transcript view based on the specified column format, defaults to False
column_case: The column_case parameter in the create_transcript_view_from_column_format function is used to specify the case transformation to be applied to the columns in the transcript view. It can be set to either "upper" or "lower" to convert the column names to uppercase or lowercase, respectively

Returns

The create_transcript_view_from_column_format function returns two lists: temporary_tables and annotation_fields.

def create_transcript_view( self, transcripts_table: str = None, transcripts_table_drop: bool = False, param: dict = {}) -> str: View Source

10860    def create_transcript_view(
10861        self,
10862        transcripts_table: str = None,
10863        transcripts_table_drop: bool = False,
10864        param: dict = {},
10865    ) -> str:
10866        """
10867        The `create_transcript_view` function generates a transcript view by processing data from a
10868        specified table based on provided parameters and structural information.
10869
10870        :param transcripts_table: The `transcripts_table` parameter in the `create_transcript_view` function
10871        is used to specify the name of the table that will store the final transcript view data. If a table
10872        name is not provided, the function will create a new table to store the transcript view data, and by
10873        default,, defaults to transcripts
10874        :type transcripts_table: str (optional)
10875        :param transcripts_table_drop: The `transcripts_table_drop` parameter in the
10876        `create_transcript_view` function is a boolean parameter that determines whether to drop the
10877        existing transcripts table before creating a new one. If `transcripts_table_drop` is set to `True`,
10878        the function will drop the existing transcripts table if it exists, defaults to False
10879        :type transcripts_table_drop: bool (optional)
10880        :param param: The `param` parameter in the `create_transcript_view` function is a dictionary that
10881        contains information needed to create a transcript view. It includes details such as the structure
10882        of the transcripts, columns mapping, column formats, and other necessary information for generating
10883        the view. This parameter allows for flexibility and customization
10884        :type param: dict
10885        :return: The `create_transcript_view` function returns the name of the transcripts table that was
10886        created or modified during the execution of the function.
10887        """
10888
10889        log.debug("Start transcripts view creation...")
10890
10891        # Default
10892        transcripts_table_default = "transcripts"
10893
10894        # Param
10895        if not param:
10896            param = self.get_param()
10897
10898        # Struct
10899        struct = param.get("transcripts", {}).get("struct", None)
10900
10901        # Transcript veresion
10902        transcript_id_remove_version = param.get("transcripts", {}).get(
10903            "transcript_id_remove_version", False
10904        )
10905
10906        # Transcripts mapping
10907        transcript_id_mapping_file = param.get("transcripts", {}).get(
10908            "transcript_id_mapping_file", None
10909        )
10910
10911        # Transcripts mapping
10912        transcript_id_mapping_force = param.get("transcripts", {}).get(
10913            "transcript_id_mapping_force", None
10914        )
10915
10916        # Transcripts table
10917        if transcripts_table is None:
10918            transcripts_table = param.get("transcripts", {}).get(
10919                "table", transcripts_table_default
10920            )
10921
10922        # Check transcripts table exists
10923        if transcripts_table:
10924
10925            # Query to check if transcripts table exists
10926            query_check_table = f"""
10927                SELECT * 
10928                FROM information_schema.tables 
10929                WHERE table_name = '{transcripts_table}'
10930            """
10931            df_check_table = self.get_query_to_df(query=query_check_table)
10932
10933            # Check if transcripts table exists
10934            if len(df_check_table) > 0 and not transcripts_table_drop:
10935                log.debug(f"Table {transcripts_table} exists and not drop option")
10936                return transcripts_table
10937
10938        if struct:
10939
10940            # added_columns
10941            added_columns = []
10942
10943            # Temporary tables
10944            temporary_tables = []
10945
10946            # Annotation fields
10947            annotation_fields = []
10948
10949            # from columns map
10950            columns_maps = struct.get("from_columns_map", [])
10951            added_columns_tmp, temporary_tables_tmp, annotation_fields_tmp = (
10952                self.create_transcript_view_from_columns_map(
10953                    transcripts_table=transcripts_table,
10954                    columns_maps=columns_maps,
10955                    added_columns=added_columns,
10956                    temporary_tables=temporary_tables,
10957                    annotation_fields=annotation_fields,
10958                )
10959            )
10960            added_columns += added_columns_tmp
10961            temporary_tables += temporary_tables_tmp
10962            annotation_fields += annotation_fields_tmp
10963
10964            # from column format
10965            column_formats = struct.get("from_column_format", [])
10966            temporary_tables_tmp, annotation_fields_tmp = (
10967                self.create_transcript_view_from_column_format(
10968                    transcripts_table=transcripts_table,
10969                    column_formats=column_formats,
10970                    temporary_tables=temporary_tables,
10971                    annotation_fields=annotation_fields,
10972                )
10973            )
10974            temporary_tables += temporary_tables_tmp
10975            annotation_fields += annotation_fields_tmp
10976
10977            # Remove some specific fields/column
10978            annotation_fields = list(set(annotation_fields))
10979            for field in ["#CHROM", "POS", "REF", "ALT", "INFO", "transcript"]:
10980                if field in annotation_fields:
10981                    annotation_fields.remove(field)
10982
10983            # Merge temporary tables query
10984            query_merge = ""
10985            for temporary_table in list(set(temporary_tables)):
10986
10987                # First temporary table
10988                if not query_merge:
10989                    query_merge = f"""
10990                        SELECT * FROM {temporary_table}
10991                    """
10992                # other temporary table (using UNION)
10993                else:
10994                    query_merge += f"""
10995                        UNION BY NAME SELECT * FROM {temporary_table}
10996                    """
10997
10998            # transcript table tmp
10999            transcript_table_tmp = "transcripts_tmp"
11000            transcript_table_tmp2 = "transcripts_tmp2"
11001            transcript_table_tmp3 = "transcripts_tmp3"
11002
11003            # Merge on transcript
11004            query_merge_on_transcripts_annotation_fields = []
11005
11006            # Add transcript list
11007            query_merge_on_transcripts_annotation_fields.append(
11008                f""" list_aggregate(list_distinct(array_agg({transcript_table_tmp}.transcript)), 'string_agg', ',') AS transcript_list """
11009            )
11010
11011            # Aggregate all annotations fields
11012            for annotation_field in set(annotation_fields):
11013                query_merge_on_transcripts_annotation_fields.append(
11014                    f""" list_aggregate(list_distinct(array_agg({transcript_table_tmp}.{annotation_field})), 'string_agg', ',') AS {annotation_field} """
11015                )
11016
11017            # Transcripts mapping
11018            if transcript_id_mapping_file:
11019
11020                # Transcript dataframe
11021                transcript_id_mapping_dataframe_name = "transcript_id_mapping_dataframe"
11022                transcript_id_mapping_dataframe = transcripts_file_to_df(
11023                    transcript_id_mapping_file, column_names=["transcript", "alias"]
11024                )
11025
11026                # Transcript version remove
11027                if transcript_id_remove_version:
11028                    query_transcript_column_select = f"split_part({transcript_table_tmp}.transcript, '.', 1) AS transcript_original, split_part({transcript_id_mapping_dataframe_name}.transcript, '.', 1) AS transcript_mapped"
11029                    query_transcript_column_group_by = f"split_part({transcript_table_tmp}.transcript, '.', 1), split_part({transcript_id_mapping_dataframe_name}.transcript, '.', 1)"
11030                    query_left_join = f"""
11031                        LEFT JOIN {transcript_id_mapping_dataframe_name} ON (split_part({transcript_id_mapping_dataframe_name}.alias, '.', 1)=split_part({transcript_table_tmp}.transcript, '.', 1))
11032                    """
11033                else:
11034                    query_transcript_column_select = f"{transcript_table_tmp}.transcript AS transcript_original, {transcript_id_mapping_dataframe_name}.transcript AS transcript_mapped"
11035                    query_transcript_column_group_by = f"{transcript_table_tmp}.transcript, {transcript_id_mapping_dataframe_name}.transcript"
11036                    query_left_join = f"""
11037                        LEFT JOIN {transcript_id_mapping_dataframe_name} ON (split_part({transcript_id_mapping_dataframe_name}.alias, '.', 1)=split_part({transcript_table_tmp}.transcript, '.', 1))
11038                    """
11039
11040                # Transcript column for group by merge
11041                query_transcript_merge_group_by = """
11042                        CASE
11043                            WHEN transcript_mapped NOT IN ('')
11044                            THEN split_part(transcript_mapped, '.', 1)
11045                            ELSE split_part(transcript_original, '.', 1)
11046                        END
11047                    """
11048
11049                # Merge query
11050                transcripts_tmp2_query = f"""
11051                    SELECT "#CHROM", POS, REF, ALT, INFO, {query_transcript_column_select}, {", ".join(query_merge_on_transcripts_annotation_fields)}
11052                    FROM ({query_merge}) AS {transcript_table_tmp}
11053                    {query_left_join}
11054                    GROUP BY "#CHROM", POS, REF, ALT, INFO, {query_transcript_column_group_by}
11055                """
11056
11057                # Retrive columns after mege
11058                transcripts_tmp2_describe_query = f"""
11059                    DESCRIBE {transcripts_tmp2_query}
11060                """
11061                transcripts_tmp2_describe_list = list(
11062                    self.get_query_to_df(query=transcripts_tmp2_describe_query)[
11063                        "column_name"
11064                    ]
11065                )
11066
11067                # Create list of columns for select clause
11068                transcripts_tmp2_describe_select_clause = []
11069                for field in transcripts_tmp2_describe_list:
11070                    if field not in [
11071                        "#CHROM",
11072                        "POS",
11073                        "REF",
11074                        "ALT",
11075                        "INFO",
11076                        "transcript_mapped",
11077                    ]:
11078                        as_field = field
11079                        if field in ["transcript_original"]:
11080                            as_field = "transcripts_mapped"
11081                        transcripts_tmp2_describe_select_clause.append(
11082                            f""" list_aggregate(list_distinct(array_agg({transcript_table_tmp2}.{field})), 'string_agg', ',') AS {as_field} """
11083                        )
11084
11085                # Merge with mapping
11086                query_merge_on_transcripts = f"""
11087                    SELECT
11088                        "#CHROM", POS, REF, ALT, INFO,
11089                        CASE
11090                            WHEN ANY_VALUE(transcript_mapped) NOT IN ('')
11091                            THEN ANY_VALUE(transcript_mapped)
11092                            ELSE ANY_VALUE(transcript_original)
11093                        END AS transcript,
11094                        {", ".join(transcripts_tmp2_describe_select_clause)}
11095                    FROM ({transcripts_tmp2_query}) AS {transcript_table_tmp2}
11096                    GROUP BY "#CHROM", POS, REF, ALT, INFO,
11097                        {query_transcript_merge_group_by}
11098                """
11099
11100                # Add transcript filter from mapping file
11101                if transcript_id_mapping_force:
11102                    query_merge_on_transcripts = f"""
11103                        SELECT *
11104                        FROM ({query_merge_on_transcripts}) AS {transcript_table_tmp3}
11105                        WHERE split_part({transcript_table_tmp3}.transcript, '.', 1) in (SELECT split_part(transcript, '.', 1) FROM transcript_id_mapping_dataframe)
11106                    """
11107
11108            # No transcript mapping
11109            else:
11110
11111                # Remove transcript version
11112                if transcript_id_remove_version:
11113                    query_transcript_column = f"""
11114                        split_part({transcript_table_tmp}.transcript, '.', 1)
11115                    """
11116                else:
11117                    query_transcript_column = """
11118                        transcript
11119                    """
11120
11121                # Query sections
11122                query_transcript_column_select = (
11123                    f"{query_transcript_column} AS transcript"
11124                )
11125                query_transcript_column_group_by = query_transcript_column
11126
11127                # Query for transcripts view
11128                query_merge_on_transcripts = f"""
11129                    SELECT "#CHROM", POS, REF, ALT, INFO, {query_transcript_column} AS transcript, NULL AS transcript_mapped, {", ".join(query_merge_on_transcripts_annotation_fields)}
11130                    FROM ({query_merge}) AS {transcript_table_tmp}
11131                    GROUP BY "#CHROM", POS, REF, ALT, INFO, {query_transcript_column}
11132                """
11133
11134            # Drop transcript view is necessary
11135            if transcripts_table_drop:
11136                query_drop = f"""
11137                    DROP TABLE IF EXISTS {transcripts_table};
11138                """
11139                self.execute_query(query=query_drop)
11140
11141            # # Merge and create transcript view
11142            # query_create_view = f"""
11143            #     CREATE TABLE IF NOT EXISTS {transcripts_table}
11144            #     AS {query_merge_on_transcripts}
11145            # """
11146            # self.execute_query(query=query_create_view)
11147
11148            # Using #CHROM chunk
11149            ######
11150
11151            # List of unique #CHROM
11152            query_unique_chrom = f"""
11153                SELECT DISTINCT "#CHROM"
11154                FROM variants AS subquery
11155            """
11156            unique_chroms = self.get_query_to_df(query=query_unique_chrom)
11157
11158            # Create table with structure but without data, if not exists
11159            query_create_table = f"""
11160                CREATE TABLE IF NOT EXISTS {transcripts_table} AS
11161                SELECT * FROM ({query_merge_on_transcripts}) AS subquery LIMIT 0
11162            """
11163            self.execute_query(query=query_create_table)
11164
11165            # Process by #CHROM
11166            for chrom in unique_chroms["#CHROM"]:
11167
11168                # Log
11169                log.debug(f"Processing #CHROM={chrom}")
11170
11171                # Select data by #CHROM
11172                query_chunk = f"""
11173                    SELECT *
11174                    FROM ({query_merge_on_transcripts})
11175                    WHERE "#CHROM" = '{chrom}'
11176                """
11177
11178                # Insert data
11179                query_insert_chunk = f"""
11180                    INSERT INTO {transcripts_table}
11181                    {query_chunk}
11182                """
11183                self.execute_query(query=query_insert_chunk)
11184
11185            # Remove temporary tables
11186            if temporary_tables:
11187                for temporary_table in list(set(temporary_tables)):
11188                    query_drop_tmp_table = f"""
11189                        DROP TABLE IF EXISTS {temporary_table}
11190                    """
11191                    self.execute_query(query=query_drop_tmp_table)
11192
11193            # Remove added columns
11194            for added_column in added_columns:
11195                self.drop_column(column=added_column)
11196
11197        else:
11198
11199            transcripts_table = None
11200
11201        return transcripts_table

The create_transcript_view function generates a transcript view by processing data from a specified table based on provided parameters and structural information.

Parameters

transcripts_table: The transcripts_table parameter in the create_transcript_view function is used to specify the name of the table that will store the final transcript view data. If a table name is not provided, the function will create a new table to store the transcript view data, and by default,, defaults to transcripts
transcripts_table_drop: The transcripts_table_drop parameter in the create_transcript_view function is a boolean parameter that determines whether to drop the existing transcripts table before creating a new one. If transcripts_table_drop is set to True, the function will drop the existing transcripts table if it exists, defaults to False
param: The param parameter in the create_transcript_view function is a dictionary that contains information needed to create a transcript view. It includes details such as the structure of the transcripts, columns mapping, column formats, and other necessary information for generating the view. This parameter allows for flexibility and customization

Returns

The create_transcript_view function returns the name of the transcripts table that was created or modified during the execution of the function.

def annotation_format_to_table( self, uniquify: bool = True, annotation_field: str = 'ANN', annotation_id: str = 'Feature_ID', view_name: str = 'transcripts', column_rename: dict = {}, column_clean: bool = False, column_case: str = None) -> str: View Source

11203    def annotation_format_to_table(
11204        self,
11205        uniquify: bool = True,
11206        annotation_field: str = "ANN",
11207        annotation_id: str = "Feature_ID",
11208        view_name: str = "transcripts",
11209        column_rename: dict = {},
11210        column_clean: bool = False,
11211        column_case: str = None,
11212    ) -> str:
11213        """
11214        The `annotation_format_to_table` function converts annotation data from a VCF file into a
11215        structured table format, ensuring unique values and creating a temporary table for further
11216        processing or analysis.
11217
11218        :param uniquify: The `uniquify` parameter is a boolean flag that determines whether to ensure
11219        unique values in the output or not. If set to `True`, the function will make sure that the
11220        output values are unique, defaults to True
11221        :type uniquify: bool (optional)
11222        :param annotation_field: The `annotation_field` parameter refers to the field in the VCF file
11223        that contains the annotation information for each variant. This field is used to extract the
11224        annotation details for further processing in the function. By default, it is set to "ANN",
11225        defaults to ANN
11226        :type annotation_field: str (optional)
11227        :param annotation_id: The `annotation_id` parameter in the `annotation_format_to_table` method
11228        is used to specify the identifier for the annotation feature. This identifier will be used as a
11229        column name in the resulting table or view that is created based on the annotation data. It
11230        helps in uniquely identifying each annotation entry in the, defaults to Feature_ID
11231        :type annotation_id: str (optional)
11232        :param view_name: The `view_name` parameter in the `annotation_format_to_table` method is used
11233        to specify the name of the temporary table that will be created to store the transformed
11234        annotation data. This table will hold the extracted information from the annotation field in a
11235        structured format for further processing or analysis. By default,, defaults to transcripts
11236        :type view_name: str (optional)
11237        :param column_rename: The `column_rename` parameter in the `annotation_format_to_table` method
11238        is a dictionary that allows you to specify custom renaming for columns. By providing key-value
11239        pairs in this dictionary, you can rename specific columns in the resulting table or view that is
11240        created based on the annotation data. This feature enables
11241        :type column_rename: dict
11242        :param column_clean: The `column_clean` parameter in the `annotation_format_to_table` method is
11243        a boolean flag that determines whether the annotation field should undergo a cleaning process.
11244        If set to `True`, the function will clean the annotation field before further processing. This
11245        cleaning step may involve removing any unwanted characters, formatting inconsistencies, defaults
11246        to False
11247        :type column_clean: bool (optional)
11248        :param column_case: The `column_case` parameter in the `annotation_format_to_table` method is
11249        used to specify the case transformation to be applied to the column names extracted from the
11250        annotation data. It allows you to set the case of the column names to either lowercase or
11251        uppercase for consistency or other specific requirements during the conversion
11252        :type column_case: str
11253        :return: The function `annotation_format_to_table` is returning the name of the view created,
11254        which is stored in the variable `view_name`.
11255        """
11256
11257        # Annotation field
11258        annotation_format = "annotation_explode"
11259
11260        # Transcript annotation
11261        if column_rename:
11262            annotation_id = column_rename.get(annotation_id, annotation_id)
11263
11264        if column_clean:
11265            annotation_id = clean_annotation_field(annotation_id)
11266
11267        # Prefix
11268        prefix = self.get_explode_infos_prefix()
11269        if prefix:
11270            prefix = "INFO/"
11271
11272        # Annotation fields
11273        annotation_infos = prefix + annotation_field
11274        annotation_format_infos = prefix + annotation_format
11275
11276        # Variants table
11277        table_variants = self.get_table_variants()
11278
11279        # Header
11280        vcf_reader = self.get_header()
11281
11282        # Add columns
11283        added_columns = []
11284
11285        # Explode HGVS field in column
11286        added_columns += self.explode_infos(fields=[annotation_field])
11287
11288        if annotation_field in vcf_reader.infos:
11289
11290            # Extract ANN header
11291            ann_description = vcf_reader.infos[annotation_field].desc
11292            pattern = r"'(.+?)'"
11293            match = re.search(pattern, ann_description)
11294            if match:
11295                ann_header_match = match.group(1).split(" | ")
11296                ann_header = []
11297                ann_header_desc = {}
11298                for i in range(len(ann_header_match)):
11299                    ann_header_info = "".join(
11300                        char for char in ann_header_match[i] if char.isalnum()
11301                    )
11302                    ann_header.append(ann_header_info)
11303                    ann_header_desc[ann_header_info] = ann_header_match[i]
11304                if not ann_header_desc:
11305                    raise ValueError("Invalid header description format")
11306            else:
11307                raise ValueError("Invalid header description format")
11308
11309            # Create variant id
11310            variant_id_column = self.get_variant_id_column()
11311            added_columns += [variant_id_column]
11312
11313            # Get list of #CHROM
11314            query_unique_chrom = f"""
11315                SELECT DISTINCT "#CHROM"
11316                FROM variants AS subquery
11317            """
11318            unique_chroms = self.get_query_to_df(query=query_unique_chrom)
11319
11320            # Base for database anontation format
11321            dataframe_annotation_format_base = f"""
11322                SELECT "#CHROM", POS, REF, ALT, INFO, "{variant_id_column}", "{annotation_infos}"
11323                FROM {table_variants}
11324            """
11325
11326            # Create dataframe for keys column type
11327            dataframe_annotation_format = self.get_query_to_df(
11328                f""" {dataframe_annotation_format_base} LIMIT 1000 """
11329            )
11330
11331            # Define a vectorized function to apply explode_annotation_format
11332            vectorized_explode_annotation_format = np.vectorize(
11333                lambda x: explode_annotation_format(
11334                    annotation=str(x),
11335                    uniquify=uniquify,
11336                    output_format="JSON",
11337                    prefix="",
11338                    header=list(ann_header_desc.values()),
11339                )
11340            )
11341
11342            # Assign the exploded annotations back to the dataframe
11343            dataframe_annotation_format[annotation_format_infos] = (
11344                vectorized_explode_annotation_format(
11345                    dataframe_annotation_format[annotation_infos].to_numpy()
11346                )
11347            )
11348
11349            # Find keys
11350            query_json = f"""
11351                SELECT distinct(unnest(json_keys({annotation_format}, '$.0'))) AS 'key'
11352                FROM dataframe_annotation_format;
11353            """
11354            df_keys = self.get_query_to_df(query=query_json)
11355
11356            # Check keys
11357            query_json_key = []
11358            for _, row in df_keys.iterrows():
11359
11360                # Key
11361                key = row.iloc[0]
11362                key_clean = key
11363
11364                # key rename
11365                if column_rename:
11366                    key_clean = column_rename.get(key_clean, key_clean)
11367
11368                # key clean
11369                if column_clean:
11370                    key_clean = clean_annotation_field(key_clean)
11371
11372                # Key case
11373                if column_case:
11374                    if column_case.lower() in ["lower"]:
11375                        key_clean = key_clean.lower()
11376                    elif column_case.lower() in ["upper"]:
11377                        key_clean = key_clean.upper()
11378
11379                # Type
11380                query_json_type = f"""
11381                    SELECT * 
11382                    FROM (
11383                        SELECT 
11384                            NULLIF(unnest(json_extract_string({annotation_format}, '$.*."{key}"')), '') AS '{key_clean}'
11385                        FROM
11386                            dataframe_annotation_format
11387                        )
11388                    WHERE "{key_clean}" NOT NULL AND "{key_clean}" NOT IN ('')
11389                """
11390
11391                # Get DataFrame from query
11392                df_json_type = self.get_query_to_df(query=query_json_type)
11393
11394                # Detect column type
11395                column_type = detect_column_type(df_json_type[key_clean])
11396
11397                # Free up memory
11398                del df_json_type
11399
11400                # Append
11401                query_json_key.append(
11402                    f"""NULLIF(unnest(json_extract_string({annotation_format}, '$.*."{key}"')), '')::{column_type}  AS '{prefix}{key_clean}' """
11403                )
11404
11405            # Create table with structure but without data, if not exists
11406            query_create_table = f"""
11407                CREATE TABLE IF NOT EXISTS {view_name}
11408                AS (
11409                    SELECT *, {annotation_id} AS 'transcript'
11410                    FROM (
11411                        SELECT "#CHROM", POS, REF, ALT, INFO, {",".join(query_json_key)}
11412                        FROM dataframe_annotation_format
11413                        )
11414                    LIMIT 0
11415                    );
11416            """
11417            self.execute_query(query=query_create_table)
11418
11419            # Free up memory
11420            del dataframe_annotation_format
11421
11422            # Insert data by chromosome
11423            for chrom in unique_chroms["#CHROM"]:
11424
11425                # Log
11426                log.debug(f"Processing #CHROM={chrom}")
11427
11428                # Create dataframe
11429                dataframe_annotation_format = self.get_query_to_df(
11430                    f""" {dataframe_annotation_format_base}  WHERE "#CHROM" = '{chrom}' """
11431                )
11432
11433                # Define a vectorized function to apply explode_annotation_format
11434                vectorized_explode_annotation_format = np.vectorize(
11435                    lambda x: explode_annotation_format(
11436                        annotation=str(x),
11437                        uniquify=uniquify,
11438                        output_format="JSON",
11439                        prefix="",
11440                        header=list(ann_header_desc.values()),
11441                    )
11442                )
11443
11444                # Assign the exploded annotations back to the dataframe
11445                dataframe_annotation_format[annotation_format_infos] = (
11446                    vectorized_explode_annotation_format(
11447                        dataframe_annotation_format[annotation_infos].to_numpy()
11448                    )
11449                )
11450
11451                # Insert data into tmp table
11452                query_insert_chunk = f"""
11453                    INSERT INTO {view_name}
11454                    SELECT *, {annotation_id} AS 'transcript'
11455                    FROM (
11456                        SELECT "#CHROM", POS, REF, ALT, INFO, {",".join(query_json_key)}
11457                        FROM dataframe_annotation_format
11458                        )
11459                """
11460                self.execute_query(query=query_insert_chunk)
11461
11462                # Free up memory
11463                del dataframe_annotation_format
11464
11465        else:
11466
11467            # Return None
11468            view_name = None
11469
11470        # Remove added columns
11471        for added_column in added_columns:
11472            self.drop_column(column=added_column)
11473
11474        return view_name

The annotation_format_to_table function converts annotation data from a VCF file into a structured table format, ensuring unique values and creating a temporary table for further processing or analysis.

Parameters

uniquify: The uniquify parameter is a boolean flag that determines whether to ensure unique values in the output or not. If set to True, the function will make sure that the output values are unique, defaults to True
annotation_field: The annotation_field parameter refers to the field in the VCF file that contains the annotation information for each variant. This field is used to extract the annotation details for further processing in the function. By default, it is set to "ANN", defaults to ANN
annotation_id: The annotation_id parameter in the annotation_format_to_table method is used to specify the identifier for the annotation feature. This identifier will be used as a column name in the resulting table or view that is created based on the annotation data. It helps in uniquely identifying each annotation entry in the, defaults to Feature_ID
view_name: The view_name parameter in the annotation_format_to_table method is used to specify the name of the temporary table that will be created to store the transformed annotation data. This table will hold the extracted information from the annotation field in a structured format for further processing or analysis. By default,, defaults to transcripts
column_rename: The column_rename parameter in the annotation_format_to_table method is a dictionary that allows you to specify custom renaming for columns. By providing key-value pairs in this dictionary, you can rename specific columns in the resulting table or view that is created based on the annotation data. This feature enables
column_clean: The column_clean parameter in the annotation_format_to_table method is a boolean flag that determines whether the annotation field should undergo a cleaning process. If set to True, the function will clean the annotation field before further processing. This cleaning step may involve removing any unwanted characters, formatting inconsistencies, defaults to False
column_case: The column_case parameter in the annotation_format_to_table method is used to specify the case transformation to be applied to the column names extracted from the annotation data. It allows you to set the case of the column names to either lowercase or uppercase for consistency or other specific requirements during the conversion

Returns

The function annotation_format_to_table is returning the name of the view created, which is stored in the variable view_name.

def transcript_view_to_variants( self, transcripts_table: str = None, transcripts_column_id: str = None, transcripts_info_json: str = None, transcripts_info_field_json: str = None, transcripts_info_format: str = None, transcripts_info_field_format: str = None, param: dict = {}) -> bool: View Source

11476    def transcript_view_to_variants(
11477        self,
11478        transcripts_table: str = None,
11479        transcripts_column_id: str = None,
11480        transcripts_info_json: str = None,
11481        transcripts_info_field_json: str = None,
11482        transcripts_info_format: str = None,
11483        transcripts_info_field_format: str = None,
11484        param: dict = {},
11485    ) -> bool:
11486        """
11487        The `transcript_view_to_variants` function updates a variants table with information from
11488        transcripts in JSON format.
11489
11490        :param transcripts_table: The `transcripts_table` parameter is used to specify the name of the
11491        table containing the transcripts data. If this parameter is not provided, the function will
11492        attempt to retrieve it from the `param` dictionary or use a default value of "transcripts"
11493        :type transcripts_table: str
11494        :param transcripts_column_id: The `transcripts_column_id` parameter is used to specify the
11495        column in the `transcripts_table` that contains the unique identifier for each transcript. This
11496        identifier is used to match transcripts with variants in the database
11497        :type transcripts_column_id: str
11498        :param transcripts_info_json: The `transcripts_info_json` parameter is used to specify the name
11499        of the column in the variants table where the transcripts information will be stored in JSON
11500        format. This parameter allows you to define the column in the variants table that will hold the
11501        JSON-formatted information about transcripts
11502        :type transcripts_info_json: str
11503        :param transcripts_info_field_json: The `transcripts_info_field_json` parameter is used to
11504        specify the field in the VCF header that will contain information about transcripts in JSON
11505        format. This field will be added to the VCF header as an INFO field with the specified name
11506        :type transcripts_info_field_json: str
11507        :param transcripts_info_format: The `transcripts_info_format` parameter is used to specify the
11508        format of the information about transcripts that will be stored in the variants table. This
11509        format can be used to define how the transcript information will be structured or displayed
11510        within the variants table
11511        :type transcripts_info_format: str
11512        :param transcripts_info_field_format: The `transcripts_info_field_format` parameter is used to
11513        specify the field in the VCF header that will contain information about transcripts in a
11514        specific format. This field will be added to the VCF header as an INFO field with the specified
11515        name
11516        :type transcripts_info_field_format: str
11517        :param param: The `param` parameter in the `transcript_view_to_variants` method is a dictionary
11518        that contains various configuration settings related to transcripts. It is used to provide
11519        default values for certain parameters if they are not explicitly provided when calling the
11520        method. The `param` dictionary can be passed as an argument
11521        :type param: dict
11522        :return: The function `transcript_view_to_variants` returns a boolean value. It returns `True`
11523        if the operation is successful and `False` if certain conditions are not met.
11524        """
11525
11526        msg_info_prefix = "Start transcripts view to variants annotations"
11527
11528        log.debug(f"{msg_info_prefix}...")
11529
11530        # Default
11531        transcripts_table_default = "transcripts"
11532        transcripts_column_id_default = "transcript"
11533        transcripts_info_json_default = None
11534        transcripts_info_format_default = None
11535        transcripts_info_field_json_default = None
11536        transcripts_info_field_format_default = None
11537
11538        # Param
11539        if not param:
11540            param = self.get_param()
11541
11542        # Transcripts table
11543        if transcripts_table is None:
11544            transcripts_table = param.get("transcripts", {}).get(
11545                "table", transcripts_table_default
11546            )
11547
11548        # Transcripts column ID
11549        if transcripts_column_id is None:
11550            transcripts_column_id = param.get("transcripts", {}).get(
11551                "column_id", transcripts_column_id_default
11552            )
11553
11554        # Transcripts info json
11555        if transcripts_info_json is None:
11556            transcripts_info_json = param.get("transcripts", {}).get(
11557                "transcripts_info_json", transcripts_info_json_default
11558            )
11559
11560        # Transcripts info field JSON
11561        if transcripts_info_field_json is None:
11562            transcripts_info_field_json = param.get("transcripts", {}).get(
11563                "transcripts_info_field_json", transcripts_info_field_json_default
11564            )
11565        # if transcripts_info_field_json is not None and transcripts_info_json is None:
11566        #     transcripts_info_json = transcripts_info_field_json
11567
11568        # Transcripts info format
11569        if transcripts_info_format is None:
11570            transcripts_info_format = param.get("transcripts", {}).get(
11571                "transcripts_info_format", transcripts_info_format_default
11572            )
11573
11574        # Transcripts info field FORMAT
11575        if transcripts_info_field_format is None:
11576            transcripts_info_field_format = param.get("transcripts", {}).get(
11577                "transcripts_info_field_format", transcripts_info_field_format_default
11578            )
11579        # if (
11580        #     transcripts_info_field_format is not None
11581        #     and transcripts_info_format is None
11582        # ):
11583        #     transcripts_info_format = transcripts_info_field_format
11584
11585        # Variants table
11586        table_variants = self.get_table_variants()
11587
11588        # Check info columns param
11589        if (
11590            transcripts_info_json is None
11591            and transcripts_info_field_json is None
11592            and transcripts_info_format is None
11593            and transcripts_info_field_format is None
11594        ):
11595            return False
11596
11597        # Transcripts infos columns
11598        query_transcripts_infos_columns = f"""
11599            SELECT *
11600            FROM (
11601                DESCRIBE SELECT * FROM {transcripts_table}
11602                )
11603            WHERE "column_name" NOT IN ('#CHROM', 'POS', 'REF', 'ALT', '{transcripts_column_id}')
11604        """
11605        transcripts_infos_columns = list(
11606            self.get_query_to_df(query=query_transcripts_infos_columns)["column_name"]
11607        )
11608
11609        # View results
11610        clause_select = []
11611        clause_to_json = []
11612        clause_to_format = []
11613        for field in transcripts_infos_columns:
11614            # Do not consider INFO field for export into fields
11615            if field not in ["INFO"]:
11616                clause_select.append(
11617                    f""" regexp_split_to_table(CAST("{field}" AS STRING), ',') AS '{field}' """
11618                )
11619                clause_to_json.append(f""" '{field}': "{field}" """)
11620                clause_to_format.append(f""" "{field}" """)
11621
11622        # Update
11623        update_set_json = []
11624        update_set_format = []
11625
11626        # VCF header
11627        vcf_reader = self.get_header()
11628
11629        # Transcripts to info column in JSON
11630        if transcripts_info_json:
11631
11632            # Create column on variants table
11633            self.add_column(
11634                table_name=table_variants,
11635                column_name=transcripts_info_json,
11636                column_type="JSON",
11637                default_value=None,
11638                drop=False,
11639            )
11640
11641            # Add header
11642            vcf_reader.infos[transcripts_info_json] = vcf.parser._Info(
11643                transcripts_info_json,
11644                ".",
11645                "String",
11646                "Transcripts in JSON format",
11647                "unknwon",
11648                "unknwon",
11649                self.code_type_map["String"],
11650            )
11651
11652            # Add to update
11653            update_set_json.append(
11654                f""" {transcripts_info_json}=t.{transcripts_info_json} """
11655            )
11656
11657        # Transcripts to info field in JSON
11658        if transcripts_info_field_json:
11659
11660            log.debug(f"{msg_info_prefix} - Annotation in JSON format...")
11661
11662            # Add to update
11663            update_set_json.append(
11664                f""" 
11665                    INFO = concat(
11666                            CASE
11667                                WHEN INFO NOT IN ('', '.')
11668                                THEN INFO
11669                                ELSE ''
11670                            END,
11671                            CASE
11672                                WHEN CAST(t.{transcripts_info_json} AS VARCHAR) NOT IN ('', '.')
11673                                THEN concat(
11674                                    ';{transcripts_info_field_json}=',
11675                                    t.{transcripts_info_json}
11676                                )
11677                                ELSE ''
11678                            END
11679                            )
11680                """
11681            )
11682
11683            # Add header
11684            vcf_reader.infos[transcripts_info_field_json] = vcf.parser._Info(
11685                transcripts_info_field_json,
11686                ".",
11687                "String",
11688                "Transcripts in JSON format",
11689                "unknwon",
11690                "unknwon",
11691                self.code_type_map["String"],
11692            )
11693
11694        if update_set_json:
11695
11696            # Update query
11697            query_update = f"""
11698                UPDATE {table_variants}
11699                    SET {", ".join(update_set_json)}
11700                FROM
11701                (
11702                    SELECT
11703                        "#CHROM", POS, REF, ALT,
11704                            concat(
11705                            '{{',
11706                            string_agg(
11707                                '"' || "{transcripts_column_id}" || '":' ||
11708                                to_json(json_output)
11709                            ),
11710                            '}}'
11711                            )::JSON AS {transcripts_info_json}
11712                    FROM
11713                        (
11714                        SELECT
11715                            "#CHROM", POS, REF, ALT,
11716                            "{transcripts_column_id}",
11717                            to_json(
11718                                {{{",".join(clause_to_json)}}}
11719                            )::JSON AS json_output
11720                        FROM
11721                            (SELECT "#CHROM", POS, REF, ALT, "{transcripts_column_id}", {", ".join(clause_select)} FROM {transcripts_table})
11722                        WHERE "{transcripts_column_id}" IS NOT NULL
11723                        )
11724                    GROUP BY "#CHROM", POS, REF, ALT
11725                ) AS t
11726                WHERE {table_variants}."#CHROM" = t."#CHROM"
11727                    AND {table_variants}."POS" = t."POS"
11728                    AND {table_variants}."REF" = t."REF"
11729                    AND {table_variants}."ALT" = t."ALT"
11730            """
11731
11732            self.execute_query(query=query_update)
11733
11734        # Transcripts to info column in FORMAT
11735        if transcripts_info_format:
11736
11737            # Create column on variants table
11738            self.add_column(
11739                table_name=table_variants,
11740                column_name=transcripts_info_format,
11741                column_type="VARCHAR",
11742                default_value=None,
11743                drop=False,
11744            )
11745
11746            # Add header
11747            vcf_reader.infos[transcripts_info_format] = vcf.parser._Info(
11748                transcripts_info_format,
11749                ".",
11750                "String",
11751                f"Transcripts annotations: 'transcript | {' | '.join(transcripts_infos_columns)}'",
11752                "unknwon",
11753                "unknwon",
11754                self.code_type_map["String"],
11755            )
11756
11757            # Add to update
11758            update_set_format.append(
11759                f""" {transcripts_info_format}=t.{transcripts_info_format} """
11760            )
11761
11762        else:
11763
11764            # Set variable for internal queries
11765            transcripts_info_format = "transcripts_info_format"
11766
11767        # Transcripts to info field in JSON
11768        if transcripts_info_field_format:
11769
11770            log.debug(f"{msg_info_prefix} - Annotation in structured format...")
11771
11772            # Add to update
11773            update_set_format.append(
11774                f""" 
11775                    INFO = concat(
11776                            CASE
11777                                WHEN INFO NOT IN ('', '.')
11778                                THEN INFO
11779                                ELSE ''
11780                            END,
11781                            CASE
11782                                WHEN CAST(t.{transcripts_info_format} AS VARCHAR) NOT IN ('', '.')
11783                                THEN concat(
11784                                    ';{transcripts_info_field_format}=',
11785                                    t.{transcripts_info_format}
11786                                )
11787                                ELSE ''
11788                            END
11789                            )
11790                """
11791            )
11792
11793            # Add header
11794            vcf_reader.infos[transcripts_info_field_format] = vcf.parser._Info(
11795                transcripts_info_field_format,
11796                ".",
11797                "String",
11798                f"Transcripts annotations: 'transcript | {' | '.join(transcripts_infos_columns)}'",
11799                "unknwon",
11800                "unknwon",
11801                self.code_type_map["String"],
11802            )
11803
11804        if update_set_format:
11805
11806            # Update query
11807            query_update = f"""
11808                UPDATE {table_variants}
11809                    SET {", ".join(update_set_format)}
11810                FROM
11811                (
11812                    SELECT
11813                        "#CHROM", POS, REF, ALT,
11814                            string_agg({transcripts_info_format}) AS {transcripts_info_format}
11815                    FROM 
11816                        (
11817                        SELECT
11818                            "#CHROM", POS, REF, ALT,
11819                            "{transcripts_column_id}",
11820                            concat(
11821                                "{transcripts_column_id}",
11822                                '|',
11823                                {", '|', ".join(clause_to_format)}
11824                            ) AS {transcripts_info_format}
11825                        FROM
11826                            (SELECT "#CHROM", POS, REF, ALT, "{transcripts_column_id}", {", ".join(clause_select)} FROM {transcripts_table})
11827                        )
11828                    GROUP BY "#CHROM", POS, REF, ALT
11829                ) AS t
11830                WHERE {table_variants}."#CHROM" = t."#CHROM"
11831                    AND {table_variants}."POS" = t."POS"
11832                    AND {table_variants}."REF" = t."REF"
11833                    AND {table_variants}."ALT" = t."ALT"
11834            """
11835
11836            self.execute_query(query=query_update)
11837
11838        return True

The transcript_view_to_variants function updates a variants table with information from transcripts in JSON format.

Parameters

transcripts_table: The transcripts_table parameter is used to specify the name of the table containing the transcripts data. If this parameter is not provided, the function will attempt to retrieve it from the param dictionary or use a default value of "transcripts"
transcripts_column_id: The transcripts_column_id parameter is used to specify the column in the transcripts_table that contains the unique identifier for each transcript. This identifier is used to match transcripts with variants in the database
transcripts_info_json: The transcripts_info_json parameter is used to specify the name of the column in the variants table where the transcripts information will be stored in JSON format. This parameter allows you to define the column in the variants table that will hold the JSON-formatted information about transcripts
transcripts_info_field_json: The transcripts_info_field_json parameter is used to specify the field in the VCF header that will contain information about transcripts in JSON format. This field will be added to the VCF header as an INFO field with the specified name
transcripts_info_format: The transcripts_info_format parameter is used to specify the format of the information about transcripts that will be stored in the variants table. This format can be used to define how the transcript information will be structured or displayed within the variants table
transcripts_info_field_format: The transcripts_info_field_format parameter is used to specify the field in the VCF header that will contain information about transcripts in a specific format. This field will be added to the VCF header as an INFO field with the specified name
param: The param parameter in the transcript_view_to_variants method is a dictionary that contains various configuration settings related to transcripts. It is used to provide default values for certain parameters if they are not explicitly provided when calling the method. The param dictionary can be passed as an argument

Returns

The function transcript_view_to_variants returns a boolean value. It returns True if the operation is successful and False if certain conditions are not met.

def rename_info_fields(self, fields_to_rename: dict = None, table: str = None) -> dict: View Source

11840    def rename_info_fields(
11841        self, fields_to_rename: dict = None, table: str = None
11842    ) -> dict:
11843        """
11844        The `rename_info_fields` function renames specified fields in a VCF file header and updates
11845        corresponding INFO fields in the variants table.
11846
11847        :param fields_to_rename: The `fields_to_rename` parameter is a dictionary that contains the
11848        mapping of fields to be renamed in a VCF (Variant Call Format) file. The keys in the dictionary
11849        represent the original field names that need to be renamed, and the corresponding values
11850        represent the new names to which the fields should be
11851        :type fields_to_rename: dict
11852        :param table: The `table` parameter in the `rename_info_fields` function represents the name of
11853        the table in which the variants data is stored. This table contains information about genetic
11854        variants, and the function updates the corresponding INFO fields in this table when renaming
11855        specified fields in the VCF file header
11856        :type table: str
11857        :return: The `rename_info_fields` function returns a dictionary `fields_renamed` that contains
11858        the original field names as keys and their corresponding new names (or None if the field was
11859        removed) as values after renaming or removing specified fields in a VCF file header and updating
11860        corresponding INFO fields in the variants table.
11861        """
11862
11863        # Init
11864        fields_renamed = {}
11865        config = self.get_config()
11866        access = config.get("access")
11867
11868        if table is None:
11869            table = self.get_table_variants()
11870
11871        # regexp replace fonction
11872        regex_replace_dict = {}
11873        regex_replace_nb = 0
11874        regex_replace_partition = 125
11875        regex_replace = "concat(INFO, ';')"  # Add ';' to reduce regexp comlexity
11876
11877        if fields_to_rename is not None and access not in ["RO"]:
11878
11879            log.info("Rename or remove fields...")
11880
11881            # Header
11882            header = self.get_header()
11883
11884            for field_to_rename, field_renamed in fields_to_rename.items():
11885
11886                if field_to_rename in header.infos:
11887
11888                    # Rename header
11889                    if field_renamed is not None:
11890                        header.infos[field_renamed] = vcf.parser._Info(
11891                            field_renamed,
11892                            header.infos[field_to_rename].num,
11893                            header.infos[field_to_rename].type,
11894                            header.infos[field_to_rename].desc,
11895                            header.infos[field_to_rename].source,
11896                            header.infos[field_to_rename].version,
11897                            header.infos[field_to_rename].type_code,
11898                        )
11899                    del header.infos[field_to_rename]
11900
11901                    # Rename INFO patterns
11902                    field_pattern = rf"(^|;)({field_to_rename})(=[^;]*)?;"
11903                    if field_renamed is not None:
11904                        field_renamed_pattern = rf"\1{field_renamed}\3;"
11905                    else:
11906                        field_renamed_pattern = r"\1"
11907
11908                    # regexp replace
11909                    regex_replace_nb += 1
11910                    regex_replace_key = math.floor(
11911                        regex_replace_nb / regex_replace_partition
11912                    )
11913                    if (regex_replace_nb % regex_replace_partition) == 0:
11914                        regex_replace = "concat(INFO, ';')"
11915                    regex_replace = f"regexp_replace({regex_replace}, '{field_pattern}', '{field_renamed_pattern}')"
11916                    regex_replace_dict[regex_replace_key] = regex_replace
11917
11918                    # Return
11919                    fields_renamed[field_to_rename] = field_renamed
11920
11921                    # Log
11922                    if field_renamed is not None:
11923                        log.info(
11924                            f"Rename or remove fields - field '{field_to_rename}' renamed to '{field_renamed}'"
11925                        )
11926                    else:
11927                        log.info(
11928                            f"Rename or remove fields - field '{field_to_rename}' removed"
11929                        )
11930
11931                else:
11932
11933                    log.warning(
11934                        f"Rename or remove fields - field '{field_to_rename}' not in header"
11935                    )
11936
11937            # Rename INFO
11938            for regex_replace_key, regex_replace in regex_replace_dict.items():
11939                log.info(
11940                    f"Rename or remove fields - Process [{regex_replace_key+1}/{len(regex_replace_dict)}]..."
11941                )
11942                query = f"""
11943                    UPDATE {table}
11944                    SET
11945                        INFO = regexp_replace({regex_replace}, ';$', '')
11946                """
11947                log.debug(f"query={query}")
11948                self.execute_query(query=query)
11949
11950        return fields_renamed

The rename_info_fields function renames specified fields in a VCF file header and updates corresponding INFO fields in the variants table.

Parameters

fields_to_rename: The fields_to_rename parameter is a dictionary that contains the mapping of fields to be renamed in a VCF (Variant Call Format) file. The keys in the dictionary represent the original field names that need to be renamed, and the corresponding values represent the new names to which the fields should be
table: The table parameter in the rename_info_fields function represents the name of the table in which the variants data is stored. This table contains information about genetic variants, and the function updates the corresponding INFO fields in this table when renaming specified fields in the VCF file header

Returns

The rename_info_fields function returns a dictionary fields_renamed that contains the original field names as keys and their corresponding new names (or None if the field was removed) as values after renaming or removing specified fields in a VCF file header and updating corresponding INFO fields in the variants table.

def calculation_rename_info_fields( self, fields_to_rename: dict = None, table: str = None, operation_name: str = 'RENAME_INFO_FIELDS') -> None: View Source

11952    def calculation_rename_info_fields(
11953        self,
11954        fields_to_rename: dict = None,
11955        table: str = None,
11956        operation_name: str = "RENAME_INFO_FIELDS",
11957    ) -> None:
11958        """
11959        The `calculation_rename_info_fields` function retrieves parameters from a dictionary, updates
11960        fields to rename and table if provided, and then calls another function to rename the fields.
11961
11962        :param fields_to_rename: `fields_to_rename` is a dictionary that contains the fields to be
11963        renamed in a table. Each key-value pair in the dictionary represents the original field name as
11964        the key and the new field name as the value
11965        :type fields_to_rename: dict
11966        :param table: The `table` parameter in the `calculation_rename_info_fields` method is used to
11967        specify the name of the table for which the fields are to be renamed. It is a string type
11968        parameter
11969        :type table: str
11970        :param operation_name: The `operation_name` parameter in the `calculation_rename_info_fields`
11971        method is a string that specifies the name of the operation being performed. In this context, it
11972        is used as a default value for the operation name if not explicitly provided when calling the
11973        function, defaults to RENAME_INFO_FIELDS
11974        :type operation_name: str (optional)
11975        """
11976
11977        # Param
11978        param = self.get_param()
11979
11980        # Get param fields to rename
11981        param_fields_to_rename = (
11982            param.get("calculation", {})
11983            .get("calculations", {})
11984            .get(operation_name, {})
11985            .get("fields_to_rename", None)
11986        )
11987
11988        # Get param table
11989        param_table = (
11990            param.get("calculation", {})
11991            .get("calculations", {})
11992            .get(operation_name, {})
11993            .get("table", None)
11994        )
11995
11996        # Init fields_to_rename
11997        if fields_to_rename is None:
11998            fields_to_rename = param_fields_to_rename
11999
12000        # Init table
12001        if table is None:
12002            table = param_table
12003
12004        renamed_fields = self.rename_info_fields(
12005            fields_to_rename=fields_to_rename, table=table
12006        )
12007
12008        log.debug(f"renamed_fields:{renamed_fields}")

The calculation_rename_info_fields function retrieves parameters from a dictionary, updates fields to rename and table if provided, and then calls another function to rename the fields.

Parameters

fields_to_rename: fields_to_rename is a dictionary that contains the fields to be renamed in a table. Each key-value pair in the dictionary represents the original field name as the key and the new field name as the value
table: The table parameter in the calculation_rename_info_fields method is used to specify the name of the table for which the fields are to be renamed. It is a string type parameter
operation_name: The operation_name parameter in the calculation_rename_info_fields method is a string that specifies the name of the operation being performed. In this context, it is used as a default value for the operation name if not explicitly provided when calling the function, defaults to RENAME_INFO_FIELDS